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PROGRAM  EVALUATION  METRICS  FOR  U.S.  ARMY  LIFELONG  LEARNING  CENTERS 


EXECUTIVE  SUMMARY 


Research  Requirement: 

The  same  factors  that  create  a  greater  need  for  Soldiers  and  leaders  to  have  education  and 
training  that  will  help  them  to  learn  and  adapt  quickly-e.g.,  frequent  tours  of  duty  and  rapidly 
changing  equipment  and  technology  requirements— make  it  more  difficult  for  Soldiers  and 
leaders  to  receive  the  instruction  they  need  via  the  traditional  Army  education  system.  The 
lifelong  learning  concept,  launched  by  the  U.S.  Army  Signal  Center,  has  been  advanced  by  the 
U.S.  Army  Training  and  Doctrine  Command  as  an  Army-wide  solution  to  this  problem.  Lifelong 
Learning  Centers  (LLCs)  comprise  a  suite  of  technologies  that  enable,  among  other  things, 
online  posting  of  schoolhouse  curriculum  materials  and  synchronous  and  asynchronous 
collaboration  among  students,  instructors,  curriculum  developers,  and  other  users.  These 
technologies  connect  the  field  Army  to  Army  schoolhouses,  simultaneously  improving  course 
currency  and  supporting  training  in  the  field  through  just-in-time  reachback  to  the  institution. 

The  impact  of  lifelong  learning  on  organizational  excellence  appears  clear— mission 
readiness  will  be  enhanced  over  the  short  and  long  term,  with  obvious  implications  for 
operational  success.  What  is  less  well  understood  is  how  LLCs  promote  readiness  through  the 
use  of  educational  technology  and  how  the  effectiveness  of  LLCs  should  be  captured.  The 
purpose  of  the  present  investigation  was  to  research  and  develop  a  comprehensive,  generalizable 
framework  for  conceptualizing  the  effectiveness  of  LLCs  in  enhancing  learning  and  readiness 
and  for  capturing  the  drivers  of  LLC  success.  The  goal  was  to  produce  an  assessment  framework 
and  associated  metrics  that  would  be  applicable,  with  minor  modification,  across  current  and 
future  LLCs. 

Procedure: 

Through  a  combination  of  scholarly  and  technical  literature  review  and  face-to-face 
interviews,  phone  conversations,  and  email  exchanges  with  key  stakeholders,  an  assessment 
framework  based  on  the  logic  modeling  (or  causal)  approach  and  blended  learning  theory  was 
produced.  The  framework  and  associated  metrics  developed  were  used  to  conduct  a  formative 
evaluation  of  an  LLC  currently  being  piloted  at  the  Fort  Leavenworth  Command  and  General 
Staff  College. 

Findings: 

The  assessment  framework  proved  to  be  a  useful  tool  for  capturing  and  representing  the 
functioning  of  blended  learning  initiatives  such  as  LLCs,  from  the  acquisition  and  use  of 
resources  to  the  achievement  of  organizational  impact.  Furthermore,  a  review  of  the  assessment 
framework  by  key  stakeholders  from  each  of  the  pilot  LLCs  ensured  generalizability  across  the 
different  initiatives.  The  findings  of  the  Fort  Leavenworth  evaluation  indicated  the  importance  of 
taking  a  causal  approach.  An  assessment  of  outcomes  alone  would  have  indicated  that  the 
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initiative  had  achieved  its  goals  but  would  have  obscured  the  fact  that  a  subset  of  these  goals— 
teaching  and  learning  effectiveness— was  achieved  largely  independently  of  the  use  of  blended 
learning  technologies. 

Utilization  and  Dissemination  of  Findings: 

The  assessment  and  evaluation  of  LLC  effectiveness  is  necessary  to  justify  continued 
investment  in  lifelong  learning.  Moreover,  the  optimal  time  to  begin  assessing  LLCs  is  now, 
during  the  initial  stages  of  their  implementation.  A  causal  assessment  framework  such  as  the  one 
researched  and  developed  in  the  present  evaluation  verifies  whether  the  apparent  success  of  an 
LLC  is  truly  driven  by  the  activities  and  outputs  generated  by  the  initiative.  It  also  provides  clear 
recommendations  for  bridging  gaps  between  resources,  personnel  activities,  system  outputs,  and 
outcomes  in  order  to  enhance  impact.  The  basis  of  this  framework  in  the  theory  of  adult  learning 
and  blended  learning  methodology  makes  it  generalizable  not  only  across  current  and  future 
LLCs,  but  also  across  blended  learning  initiatives  more  generally,  addressing  a  gap  in  the 
scholarly  literature  with  regard  to  the  effectiveness  assessment  of  educational  technology. 
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INTRODUCTION 


The  U.S.  Army’s  contemporary  operating  environment  is  characterized  by  frequent  tours 
of  duty,  heavy  reliance  on  the  Reserves  and  the  National  Guard,  the  introduction  of  new 
equipment  and  technology  immediately  prior  to  and  during  deployment,  and  a  volatile  global 
political  situation.  All  of  these  factors  result  in  a  greater  need  for  Soldiers  and  leaders  to  obtain 
the  education  and  training  that  will  help  them  to  learn  rapidly  and  adapt  quickly.  Yet  these  same 
conditions  also  make  it  more  difficult  for  Soldiers  and  leaders  to  receive  the  instruction  they 
need  via  the  traditional  Army  education  system. 

The  lifelong  learning  concept,  launched  by  the  U.S.  Army  Signal  Center’s  University  of 
Information  Technology,  has  been  advanced  by  the  U.S.  Army  Training  and  Doctrine  Command 
(TRADOC)  as  an  Army-wide  solution  to  this  problem  (TRADOC,  2004).  Lifelong  learning  has 
been  defined  as  “the  ability  of  Soldiers  to  learn,  grow  and  achieve  technically  and  tactically 
throughout  their  career,  wherever  they  serve. . .  It’s  a  mixture  of  traditional  schoolhouse  resident 
education  with  education  presented  in  other  locations  at  the  individual’s  teachable  moment” 
(TRADOC,  2004).  Lifelong  Learning  Centers  (LLCs)  comprise  a  suite  of  technologies  that 
enable,  among  other  things,  online  posting  of  schoolhouse  curriculum  materials,  synchronous 
and  asynchronous  collaboration  among  students,  instructors,  curriculum  developers,  and  other 
users,  and  single-source  course  administration  and  learning  management.  These  technologies 
connect  the  field  Army  to  Army  schoolhouses,  simultaneously  improving  course  currency  and 
supporting  training  in  the  field  through  just-in-time  reachback  capability.  Two  LLCs,  in  addition 
to  the  University  of  Information  Technology,  currently  are  being  piloted  at  Forts  Leavenworth 
and  Leonard  Wood,  with  future  LLCs  envisioned  at  the  other  TRADOC  installations. 

The  impact  of  lifelong  learning  on  organizational  excellence  seems  clear— mission 
readiness  should  be  enhanced  over  the  short  and  intermediate  term,  with  obvious  implications  for 
operational  success.  Over  the  long  term,  lifelong  learning  should  foster  a  cultural  shift  in  the 
Army  toward  collaboration,  information  sharing,  and  knowledge  management.  What  is  less  well 
understood  is  how  the  enablers  of  lifelong  learning,  LLCs,  promote  readiness  and  cultural  shift 
through  the  use  of  educational  technology  and  how  the  effectiveness  of  LLCs  should  be 
captured.  Various  approaches  to  assessing  the  costs  and  benefits  of  technology-assisted  learning 
have  been  advanced  (e.g.,  Cukier,  1997;  Ehrmann,  1994;  Whalen  &  Wright,  1999),  but  none 
have  provided  the  complete  picture  of  blended  learning  effectiveness  required  to  diagnose 
problems  and  make  specific  recommendations  for  change.  For  example,  where  assessment 
approaches  were  explicitly  specified,  they  often  targeted  either  the  cost  or  benefit  of  an 
educational  initiative,  but  not  both  (Cukier,  1997).  Some  assessment  approaches  have 
emphasized  a  systems  approach  to  conceptualizing  an  educational  initiative  (as  opposed  to 
recognizing  strictly  its  outcomes),  but  yet  have  excluded  some  components  of  the  system  that 
drive  effectiveness  (e.g.,  Ehrmann,  1994). 

Assessment  and  evaluation  of  LLC  effectiveness  is  necessary  to  justify  continued 
investment  in  lifelong  learning.  Moreover,  the  optimal  time  to  begin  assessing  LLCs  is  now, 
during  the  initial  stages  of  their  implementation  (Clark,  1994;  Sims,  Dobbs,  &  Hand,  2002).  The 
purpose  of  the  present  investigation  was  to  research  and  develop  a  comprehensive,  generalizable 
framework  for  conceptualizing  the  effectiveness  of  LLCs  in  enhancing  learning  and  readiness 
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and  for  capturing  the  drivers  of  LLC  success.  The  goal  was  to  produce  an  assessment  framework 
and  associated  metrics  that  would  be  applicable,  with  minor  modification,  across  current  and 
future  LLCs.  The  framework  and  metrics  developed  in  this  research  were  used  to  conduct  a 
formative  evaluation  of  the  Fort  Leavenworth  pilot  LLC.  The  results  of  this  evaluation  provide 
guidance  for  the  establishment  and  implementation  of  future  LLCs. 

WHAT  TO  EVALUATE?  OVERVIEW  OF  THE  APPROACH 

Educational  program  evaluation  primarily  informs  two  decisions:  (1)  whether  to  continue 
funding  for  a  particular  initiative;  and  (2)  where  to  invest  resources  (i.e.,  money,  people,  and/or 
technology,  equipment,  and  supplies)  in  order  to  facilitate  an  ongoing  initiative’s  effectiveness 
(Champagne  &  Wisher,  2000).  To  support  decision-making  in  the  former  situation,  the 
evaluation  must  assess  program  outcomes  relative  to  program  goals.  In  the  latter  case,  the 
evaluation  not  only  must  assess  goal  achievement;  it  also  must  capture  the  causal  factors  driving 
the  assessment  results.  Evaluation  of  a  program’s  drivers  of  success— in  addition  to  its  impact— 
tells  a  story  about  program  functioning  and  identifies  candidate  causes  for  shortfalls  in 
effectiveness  and  providing  the  leverage  for  change  (see  Cianciolo,  Heiden,  &  Prevou,  2006; 
McLaughlin  &  Jordan,  2004). 

This  second  type  of  effectiveness  evaluation  also  is  helpful  for  making  go/no-go  funding 
decisions  for  an  initiative  that  shows  mixed  goal  attainment.  Equipped  with  a  story  that  links 
program  outcomes  to  causal  factors,  decision  makers  can  determine  whether  continued  funding 
will  address  impact  shortfalls  or  whether  organizational  resources  are  best  invested  elsewhere.  It 
is  realistic  to  expect  that  most,  if  not  all,  educational  program  evaluations  will  produce  mixed 
results  with  regard  to  educational  impact,  especially  evaluations  of  programs  in  the  pilot  stage  of 
implementation. 

Because  an  evaluation  approach  that  captures  both  effects  and  causes  is  more  (a) 
informative  for  diagnosing  impact  shortfalls;  (b)  flexible  in  its  support  of  decision  making;  and 
(c)  robust  to  the  stage  of  implementation  to  which  it  is  applied,  this  type  of  approach  was 
selected  as  the  basis  for  the  LLC  evaluation  framework  developed  in  the  present  investigation. 
Background  research,  conducted  to  inform  the  overall  design  of  the  framework,  focused  on  the 
scholarly  and  professional  literature  exploring  the  impact  of  educational  technology.  Assessment 
frameworks  that  were  implicit  (e.g.,  tacit  assumptions  guiding  quasi-experimental  studies)  or 
explicit  (e.g.,  formal  theories)  were  considered. 

Background  -  Evaluation  Studies  Measuring  Effects 

Interest  in  evaluating  the  effectiveness  of  educational  technology  is  not  new.  Technology- 
assisted  education  has  been  examined  extensively  for  its  cost-effectiveness  and  learning  impact 
at  least  since  the  1970s  (e.g.,  Clark,  1994;  Cukier,  1997;  Levin,  1986;  Russell,  1999).  In  much  of 
the  scholarly  and  technical  research,  which  focused  on  comparing  traditional  and  technological 
solutions,  decisions  regarding  what  to  measure  often  had  an  implicit  or  empirical  basis  (i.e.,  as 
opposed  to  being  theory-  or  model-based).  In  addition,  these  studies  were  characterized  by  a 
singular  focus  on  effects,  or  outcomes,  such  as  learning  achievement  and/or  cost  savings 
[occasionally  also  time  savings,  as  in  Sanders  and  Burnside  (2001)]. 
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Effectiveness  Conceptualized  as  Learning  Achievement 

The  majority  of  research  exploring  the  effectiveness  of  technology-supported  instruction 
has  used  a  quasi-experimental  design  to  compare  the  achievement  of  students  enrolled  in  a 
course  using  a  specific  educational  technology  against  students  enrolled  in  the  same  course  using 
more  traditional  means  of  curriculum  delivery  (e.g.,  Barry  &  Runyan,  1995;  Gifford,  1998; 

Keene  &  Cary,  1992;  Hiltz,  1990;  Mclsaac,  Blocher,  Mahes,  &  Vrasidas,  1999;  Sanders  & 
Burnside,  2001).  Learning  effectiveness  was  demonstrated  if  students  using  educational 
technology  performed  as  well  or  better  than  students  completing  the  course  of  study  using 
traditional  means. 

The  measures  of  learning  achievement  used  in  these  studies  generally  corresponded  to  the 
first,  or  affective,  level  of  Kirkpatrick’s  (1994)  four  levels  of  evaluation.  That  is,  surveys  or 
interviews  were  used  to  capture  students’  reactions  to/satisfaction  with  the  course  they  took, 
including  their  perceptions  of  how  much  they  learned  or  how  beneficial  they  thought  the  course 
was.  Several  of  the  studies  also  compared  student  achievement  using  Level-2  measures— 
measures  of  actual  learning.  Investigators  either  used  existing  course  assessment  materials  or 
they  developed  special  purpose  measures  related  to  course  content,  such  as  mastery  tests  (e.g., 
Hiltz,  1990). 

The  large  body  of  research  demonstrating  that  technology  can  have  a  positive  (or  no 
negative)  effect  on  Level- 1  and  Level-2  measures  of  academic  achievement  generally  is  taken  as 
evidence  that  technology-assisted  education  is  effective.  A  primary  limitation  of  this  research, 
however,  is  that  it  cannot  explain  why  technology  enables  effective  learning  or  when  and  how  to 
use  technology  to  achieve  a  specific  learning  outcome  (Lockee,  Burton,  &  Cross,  1999).  Indeed, 
many  scholars  argue  that  effective  technology-assisted  learning  has  little  to  do  with  technology, 
but  instead  is  critically  dependent  on  how  technology  is  adaptively  and  selectively  applied  to 
achieving  particular  learning  objectives  (Clark,  1994;  Defense  Technical  Information  Center, 
2000;  Ehrmann,  1994;  Firdyiwek,  1999;  Hannafin  &  Land,  1997;  Owston,  1997;  Willis,  2000). 
This  possibility  could  account  for  the  fact  that  some  comparison  studies  show  mixed  or  even 
negative  effects  of  using  educational  technology.  Understanding  the  why,  when,  and  how  of 
effective  technology-assisted  education  is  necessary  for  understanding  the  drivers  of  instructional 
delivery  and  diagnosing  effectiveness  shortfalls. 

The  impact  of  technology-assisted  instruction  beyond  the  educational  setting  [i.e., 
Kirkpatrick’s  (1994)  third  and  fourth  levels  of  evaluation]  has  not  been  addressed  by  most 
empirical  studies.  Relatively  little  has  been  done  to  measure  behavior  change  systematically  once 
students  enter  or  re-enter  non-school  settings.  Organizational  indicators  of  success  influenced  by 
technology-assisted  education  of  the  workforce  also  remain  relatively  unexplored  (though  see 
Cianciolo  et  al.,  2006  for  an  exception).  This  gap  likely  is  the  result  of  the  difficulty  of  obtaining 
job  performance  data  and  organizational  measures  of  success,  although  some  educational 
program  evaluations  focus  exclusively  on  organizational  indicators  of  success,  such  as  return  on 
investment  and  cost  savings. 
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Effectiveness  Conceptualized  as  Cost  Savings 

Proponents  of  technology-supported  instruction,  especially  distance  education,  claim  that 
the  introduction  of  educational  technology  will  produce  a  significant  cost  savings  through 
reduced  faculty  and  administrative  staff  expenses,  reduced  materials  production,  storage,  and/or 
shipping  costs,  and  reduced  travel  and  student  housing  expenses.  Even  though  the  fixed  costs 
associated  with  procuring  instructional  technology  are  greater,  variable  costs  tend  to  be  lower 
when  fewer  instructors  and  printed  course  materials  are  required,  for  example  if  course  duration 
is  compressed  (see,  e.g.,  Whalen  &  Wright,  1999).  Although  the  literature  on  cost  accounting  for 
technology-assisted  education  is  quite  diffuse,  nearly  all  authors  argue  that  when  conducting 
such  cost  studies,  all  relevant  expenses  must  be  accounted  for,  including  not  simply  the  cost  of 
the  technology,  but  the  cost  to  prepare,  deliver,  and  support  the  course  (Clark,  1994;  Levin, 

1986;  Morgan,  2000). 

Relative  to  the  amount  of  discussion  on  the  cost  of  technology-supported  education,  few 
published  studies  have  been  conducted  that  actually  calculate  the  cost  savings  enabled  by 
technology-supported  instruction.  One  exception  is  the  work  of  Phelps,  Ashworth,  and  Hahn 
(1991),  who  compared  the  cost  of  delivering  the  U.S.  Army  Engineer  Officer  Advanced  Course 
via  asynchronous  computer  conferencing  to  the  cost  of  delivering  the  same  course  in  the 
schoolhouse.  The  cost  categories  they  included  were  course  production  (based  on  estimated  time 
values  and  hourly  rates  for  courseware  production),  equipment,  training,  supplies,  and  operations 
and  support.  Whalen  and  Wright  (1999)  calculated  projected  cost  differences  between  five 
different  course  delivery  methods  (five  technology-supported  and  one  traditional),  using  a 
detailed  methodology  that  included  the  full-range  of  fixed  and  variable  costs  associated  with 
course  production  and  delivery.  Their  approach  enabled  the  replicable  cost  comparison  of 
different  applications  of  instructional  technology  and  did  not  require  readers  to  assume  that 
different  types  of  technology  have  equivalent  implications  for  cost  relative  to  traditional 
instruction. 

Discussions  and  analyses  of  cost  savings  generally  focus  solely  on  models  and  methods 
for  cost  accounting  and  rarely  also  discuss  educational  benefits  (Cukier,  1997;  though  see  Phelps 
et  ah,  1991,  for  an  exception).  Cost  accounting  in  education  is  an  extremely  complex  topic  and 
assessment  of  cost  savings  requires  a  different  type  of  expertise  than  assessment  of  learning 
outcomes,  so  perhaps  for  these  reasons  it  is  not  coupled  with  educational  measurement. 

However,  assessing  financial  and  learning  impact  within  the  same  evaluation  would  enable 
decision  makers  to  make  informed  judgments  about  cost-benefit  tradeoffs  (Cukier,  1997).  The 
increased  costs  associated  with  technology-based  course  delivery  may  be  worth  the  expense  if 
educational  objectives  and/or  learning  audiences  previously  beyond  reach  could  be  obtained. 

Background  -  Evaluation  Models  Addressing  Causes  and  Effects 

Several  scholars  recognize  that  multiple  factors,  such  as  individual  learner  and/or 
instructor  readiness,  technology  usability,  curriculum  quality,  and  organizational  support  for 
continuous  learning,  play  a  role  in  the  effectiveness  of  an  educational  or  training  solution  (e.g., 
Dean,  Biner,  &  Coenen,  1996;  Salas,  Rhodenizer,  &  Bowers,  2000).  There  is  informal 
recognition  or  listing  of  these  factors  [e.g.,  Defense  Technical  Information  Center  (DTIC),  2000; 
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Willis,  2000]  and  there  exists  a  handful  of  formal  models  for  assessing  these  factors  as  they 
relate  to  the  success  of  technology-assisted  education  (see,  e.g.,  Belanger  &  Jordan,  2000; 
Ehrmann,  1994;  Harrison,  Seeman,  Behm,  Saba,  Molise,  &  Williams,  1991). 

For  example,  Harrison  et  al.  (1991)  identified,  through  a  combination  of  literature  review 
and  interviews,  four  categories  to  assess  when  evaluating  distance  education:  (1)  instruction 
(e.g.,  student-instructor  interaction);  (2)  management  (e.g.,  technical  support  and  planning);  (3) 
telecommuting  (largely  the  unique  features  of  distance  education);  and  (4)  support  (e.g., 
organizational  support  for  the  program).  Harrison  et  al.  used  these  categories  to  construct  an  89- 
item  course  evaluation  survey,  which  was  then  validated  in  an  experimental  study.  Scale  scores 
for  each  of  the  four  categories  were  moderately  correlated,  even  in  the  context  of  satisfactory  to 
high  scale  reliabilities,  suggesting  that  the  categories,  while  not  unique,  were  distinguishable. 
However,  the  researchers  only  examined  the  possible  factors  influencing  distance  learning  and 
did  not  attempt  to  capture  the  learning  outcomes  associated  with  the  course  evaluated  in  the 
study. 


Ehrmann  (1994)  described  the  planning  effort  behind  the  Flashlight  Project,  which  was 
intended  to  produce  generalizable,  targeted,  and  validated  methods  for  evaluating  technology- 
assisted  higher  education.  In  this  planning  effort,  Ehrmann  identified  the  targets  for  assessment, 
advocating  a  combined  process-outcome  approach.  The  outcomes  he  identified  were  learning 
effectiveness,  enrollment  and  retention,  and  cost  control.  Among  the  processes  Ehrmann 
identified  were  teaching-learning  strategies  especially  relevant  to  technology-supported 
instruction  and  faculty  roles  and  attitudes  towards  instructional  technology.  Ehrmann  argued  that 
improvements  in  learning  outcomes,  access,  and  costs  on  a  departmental  and/or  institutional 
scale  required  changes  in  teaching  practices  and  learning  environments,  which  in  turn  required 
changes  in  institutional  patterns  of  technology  use. 

The  previously  described  research  exploring  the  effectiveness  of  technology-assisted 
education  provides  a  firm  foundation  for  developing  metrics  to  assess  the  learning  effectiveness 
and  organizational  impact  of  Army  LLCs.  One  could  conclude  from  this  body  of  work  that  (a)  it 
is  important  to  use  multiple  levels  of  analysis  to  conceptualize  educational  outcomes;  (b)  cost 
metrics  should  capture  the  full  cost  of  technology-supported  instruction,  not  just  hardware  and 
software  expenses;  (c)  cost  effectiveness  metrics  should  enable  the  concurrent  review  of  costs 
and  benefits;  and  (d)  factors  leading  to  educational  outcomes,  such  as  organizational  support  and 
technology  usability  and  access,  should  be  measured  in  order  to  fully  understand  the  learning 
effectiveness  of  technology-assisted  education. 

What  the  previously  described  research  cannot  do  is  provide  a  comprehensive  theory  or 
framework  for  conceptualizing  how  Army  LLCs  achieves  educational  and  organizational  impact 
through  the  introduction  of  blended  learning  (i.e.,  combined  face-to-face,  individual, 
asynchronous  collaborative  and  synchronous  collaborative  learning)  solutions.  An  in-depth 
application  of  the  cause-effect  (or  causal)  approach  to  understanding  educational  impact  must 
occur.  Such  an  application  would  extend  previous  research  by  integrating  multiple  perspectives 
on  the  drivers  and  indicators  of  blended  learning  effectiveness  and  by  specifying  the  means  by 
which  blended  learning  has  an  impact  on  organizational  measures  of  success  through  behavior 
change. 
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A  Way  Forward:  The  Logic  Modeling  Approach 


Logic  modeling  is  a  tool  for  conceptualizing  how  a  program  transforms  resources  (e.g., 
funding,  personnel,  etc.)  into  external  results  through  activities,  output,  and  multiple  levels  of 
outcomes.  McLaughlin  and  Jordan  (2004)  describe  the  logic  model  as  a  “useful  advance 
organizer  for  designing  evaluation  and  performance  measurement”  (p.  7).  A  logic  model  makes 
explicit  the  often  tacitly  understood  functioning  of  a  program,  revealing  the  possibilities  for 
evaluation  and  the  targets  for  assessment  and  providing  a  systematic  means  to  approach  the 
cause-and-effect  evaluation  of  program  functioning.  Building  a  logic  model  is  analogous  to 
conducting  a  terrain  analysis;  it  reveals  the  areas  that  must  be  captured  (i.e.,  assessed)  in  order  to 
achieve  strategic  success  (i.e.,  maximally  informing  decision  making)  with  a  program  evaluation. 

Logic  models  are  especially  useful  for  planning  the  effectiveness  evaluation  of  programs 
not  easily  associated  with  traditional  retum-on-investment  metrics,  but  whose  continued  funding 
depends  on  demonstrations  of  impact  (Cianciolo  &  Prevou,  2006;  McLaughlin  &  Jordan,  2004). 
The  use  of  logic  models  does  not  preclude  a  cost-effectiveness  evaluation,  but  rather  provides  a 
way  to  consider  cost  and  benefits  together  as  interrelated  characteristics  of  the  same  initiative. 
Logic  models  also  prevent  the  focus  of  evaluation  from  becoming  too  narrowed  on  a  particular 
aspect  of  a  program,  such  as  outcomes— which  cannot  be  used  to  diagnose  problems— and 
activities— which  are  easy  to  measure  but  meaningless  when  disassociated  from  outcomes.  The 
logic  model  approach  used  in  the  present  evaluation  therefore  represents  an  extension  of  the 
work  reviewed  above  by  (a)  integrating  multiple  previous  assessment  approaches  into  a  unified, 
systematic  approach;  (b)  capturing  multiple  levels  of  outcome,  to  include  organizational  impact; 
and  (c)  assessing  both  cost-  and  learning-effectiveness  in  the  same  study. 

BUILDING  THE  LOGIC  MODEL  FOR  LLC  EVALUATION 

A  variety  of  resources  were  used  to  build  the  logic  model  of  LLC  impact.  The 
construction  process  began  with  a  review  of  the  planning  and  communications  documentation 
prepared  by  the  directors  of  the  three  pilot  lifelong  learning  initiatives.  This  documentation 
included  the  LLC  master  plan,  operational  requirements  statement,  use  case  summaries,  draft 
metrics  for  each  lifelong  learning  initiative,  and  implementation  progress  presentations.  Review 
of  this  documentation  identified  the  core  structure  of  the  LLCs— their  technical  and  staffing 
components— as  well  as  the  envisioned  impact  of  the  LLCs  on  Army  education  and  training, 
operational  perfonnance,  and  organizational  success.  Phone  conversations,  face-to-face 
interviews,  and/or  video  teleconferences  were  conducted  with  the  director  of  each  of  the  three 
initiatives  to  gather  their  explanations  of  how  the  technology  comprising  the  LLCs  would  lead  to 
the  envisioned  organizational  impact  through  enhanced  learning  and  behavior  change.  In 
addition,  phone  conversations  and  face-to-face  discussions  with  key  LLC  personnel  were 
conducted  in  order  to  identify  the  activities  of  each  and  their  expected  contribution  to  LLC 
functioning.  This  component  of  the  logic  model  construction  process  ensured  a  thorough, 
concrete  understanding  of  the  lifelong  learning  initiative  as  envisioned  by  the  Army  and 
implemented  by  the  individual  pilot  schools. 
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Concurrently,  an  extensive  review  of  the  scholarly  and  professional  literature  on 
technology-assisted  learning  was  conducted,  with  a  special  emphasis  on  theory  and  best  practice 
in  blended  learning.  The  literature  review  covered  a  broad  range  of  topics,  including  (a)  the 
anticipated  behavioral  outcomes  of  learning  in  a  technology-supported  environment;  (b)  the 
social  impact  of  including  technology,  especially  networked  computers,  in  the  learning  process; 
and  (c)  the  instructor,  student,  and  organizational  determinants  of  effective  learning.  A  review  of 
the  military  technical  and  professional  literature  was  conducted  to  more  fully  understand 
concretely  such  organizational  outcomes  of  interest  to  the  Army  as  mission  readiness  and  culture 
shift.  This  component  of  the  logic  model  construction  process  fleshed  out  the  information 
gathered  from  the  LLC  planning  and  communications  documentation  and  interviews  with  LLC 
leadership  and  staff.  It  helped  to  make  explicit  the  link  between  technology  and  organizational 
impact  that  often  is  implicit  in  discussions  of  technology-assisted  learning  in  the  Army  (e.g., 
Freeman,  2003). 

Finally,  face-to-face  interviews  and  focus  groups  were  conducted  with  curriculum 
developers,  instructors,  and  students  at  the  Command  and  General  Staff  School  at  Fort 
Leavenworth.  The  intent  of  these  conversations  was  to  capture  the  modifications  that  must  be 
made  to  the  general  logic  model  in  order  to  make  it  applicable  to  the  Leavenworth  lifelong 
learning  initiative  in  particular,  which  was  assessed  as  part  of  the  present  investigation. 

The  LLC  Logic  Model 

This  section  presents  a  detailed  overview  of  the  logic  model  developed  for  the  present 
research  and  depicted  in  Figure  1  below.  The  following  description  of  each  component  of  the 
model-resources,  activities,  outputs,  and  outcomes-is  coupled  with  the  justification  of  its 
inclusion  in  the  model. 

Resources 

The  resources  briefly  described  below  represent  the  categories  of  resources  typically 
found  in  the  logic  models  of  other  programs  (McLaughlin  &  Jordan,  2004).  Of  importance  is  the 
support  these  resources  provide  to  executing  the  program  activities.  Tracking  critical  resources 
makes  it  possible  for  evaluators  to  inform  decisions  regarding  resource  allocation  within  a 
program  by  linking  resources  to  outcomes  through  activity  and  output. 

Money.  Best  practice  in  cost  accounting  for  higher  education  indicates  that  there  are  three 
broad  categories  of  expense:  (1)  personnel,  supplies,  and  other  costs  directly  attributable  to 
course  development  and  delivery  (e.g.,  instructor  salaries  and  benefits);  (2)  personnel,  supplies, 
and  other  costs  indirectly  attributable  to  course  development  and  delivery  (e.g.,  salaries  and 
benefits  for  technical  support  personnel  and  administrative  staff);  and  (3)  costs  associated  with 
facilities  and  equipment  use  (e.g.,  heating  and  electricity)  (Hyatt,  1983;  Middaugh,  2000).  All 
financial  resources  associated  with  producing,  administering,  and  supporting  a  course  should  be 
accounted  for,  especially  for  technology-assisted  courses  in  which  the  temptation  is  to  limit  cost 
accounting  to  technology  requirements  (Clark,  1994;  Levin,  1986). 


7 


Personnel.  In  addition  to  the  LLC  leadership  and  the  technical  staff,  the  LLC  planning 
documentation  ( LLC  Operational  Requirements,  2005)  lists  several  other  key  personnel  types 
involved  in  lifelong  learning.  These  key  personnel  types  include  training  developers,  subject 
matter  experts,  instructors,  course  managers,  and  doctrine  developers  as  well  as  others  who 
benefit  from  the  program,  that  is,  students  and  general  purpose  users.  In  the  present  investigaiton, 
the  personnel  of  particular  focus  were  the  technical  staff  and  the  leadership,  for  their  critical 
role  in  the  online  presence  of  the  LLC,  and  the  instructors  and  training  developers  (i.e., 
curriculum  developers  and  CBT/WBT  courseware  production  teams),  for  their  critical  role 
in  providing 
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Figure  1 .  Logic  Model  for  Army  LLC  Impact 

learning  content.  Most  of  the  other  personnel  were  excluded  in  the  present  research  in  an  effort 
to  balance  the  current  and  future  instantiations  of  the  LLC.  These  excluded  personnel  roles 
represent  envisioned  users,  so  explicating  their  role  in  the  initiative  would  have  been  somewhat 
speculative  and  subject  to  change.  Students  and  general  purpose  users  were  not  included  in 
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personnel  because  their  primary  role  was  considered  to  be  more  analogous  to  a  consumer  than  a 
producer  in  the  program. 

Technology,  equipment,  and  supplies.  The  technology,  equipment,  and  supplies  for  the 
LLC  include  the  hardware,  software,  and  network  connections  necessary  to  create  and  sustain  the 
online  presence  of  the  initiative,  but  potentially  also  go  beyond  the  technical  footprint  to  include 
equipment  for  students  or  other  users  and  office  supplies  for  personnel. 

Logic  models  may  include  additional  resources  in  order  to  capture  program  inputs  that 
are  not  conventionally  considered  requirements  but  nevertheless  are  critical  to  program  success 
(McLaughlin  &  Jordan,  2004).  Chief  among  these  resources  is  a  needs  assessment.  The  needs 
assessment  states  in  concrete  detail  the  purpose  of  the  program— the  problems  it  will  solve, 
performance  or  behavioral  shortfalls  it  will  address,  and/or  the  ways  in  which  it  will  extend 
organizational  success.  The  needs  assessment  also  explains  why  the  program  is  the  best  solution. 
Importantly,  the  needs  assessment  provides  the  focus  for  program  activities,  aligning  them  with 
the  envisioned  outcomes  and  expected  impact  of  the  initiative  (Cianciolo  et  al.,  2006). 

Activities 

The  activities  of  the  LLC’s  technical  staff,  curriculum  developers,  instructors,  courseware 
production  teams,  and  leadership  collectively  produce  the  main  output  of  the  initiative-anytime, 
anywhere  access  to  Army  schoolhouse  curriculum  content.  The  tasks  these  types  of  personnel 
execute  were  determined  in  part  by  review  of  the  LLC  planning  documentation,  interviews  with 
LLC  personnel,  and,  where  necessary,  literature  review. 

Technical  staff.  The  technical  staff  is  the  critical  interface  between  LLC  users  and  the 
educational  technology.  As  such,  the  technical  staff  first  ensures  that  the  system  is  accessible  by 
performing  the  setup,  integration,  customization,  and  management  of  LLC  components  and 
migrating  course  content  across  LLC  components  (e.g.,  from  Sharepoint  to  Bb).  Secondly, 
the  technical  staff  fosters  the  effective  use  of  the  educational  technology  by  providing  training 
to  instructors,  curriculum  developers,  and  other  users  on  the  LLC  components  (see  also 
DTIC,  2000).  This  training  not  only  should  include  instruction  on  the  basic  system  functions  but 
also  on  the  standard  operating  procedures  that  have  been  developed  in  the  schoolhouse  to 
leverage  the  system  capabilities.  A  third  key  function  of  the  technical  staff  is  to  provide 
technical  support  to  students,  faculty,  curriculum  developers,  and  other  users,  assisting  with 
enrollment,  usability  problems,  and  other  inquiries  (e.g.,  password  changes).  In  serving  this 
function,  the  technical  staff  also  provides  support  for  answering  Field  Army  users’ 
operational  questions,  either  by  contacting  a  relevant,  local  subject  matter  expert  or  by 
facilitating  a  connection  between  users  and  more  remote  experts,  such  as  through  BCKS 
professional  forums.  The  effectiveness  of  technical  support  in  performing  these  functions  is  an 
important  determinant  of  the  success  of  technology-assisted  education  (DTIC,  2000;  Willis, 

2000) 


Curriculum  developers.  The  fundamental  role  of  the  curriculum  developer  is  to  populate 
the  LLC  with  engaging,  effective,  and  relevant  course  content.  In  order  to  enhance  teaching  and 
learning  through  the  development  of  more  relevant  curriculum  materials,  developers  must 
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leverage  the  capabilities  of  the  LLCs  to  collaboratively  generate  course  content,  even  when 
geographically  distributed.  Such  collaboration  ensures  that  curriculum  materials  are  integrated 
and  build  on  one  another  to  reinforce  key  learning  objectives.  Curriculum  developers  also  must 
update  course  content  based  on  feedback  from  instructors  and  students,  for  example  by 
leveraging  access  to  Army  Knowledge  Management  resources  (e.g.,  BCKS,  Center  for  Army 
Lessons  Learned).  Importantly,  early  adopters  of  the  LLCs’  educational  technology  must  lead 
the  development  of  standard  operating  procedures  (SOPs)  for  leveraging  the  capabilities  of 
the  LLC  components.  As  with  any  digital  system  with  multiple  users,  shared  awareness  of  how 
information  is  organized  is  critical  to  collaboration,  and  effective  procedures  for  building  this 
shared  awareness  must  come  from  the  collaborators  themselves.  Early  adopters  also  must 
mentor  late  adopters  on  system  functionalities  to  enhance  course  development  (Willis, 

2000).  Late  adopters  may  be  curriculum  developers  who  are  initially  resistant  to  using  the  system 
or  who  are  newly  hired  and  therefore  unfamiliar  with  the  collaborative  technologies  their  peers 
are  using.  Both  the  presence  of  SOPs  and  formal  mentorship  among  curriculum  developers 
represent  well-recognized  and  Army-endorsed  knowledge  management  practices  for  enhancing 
perfonnance. 

Instructors.  Instructors  ensure  that  students  and  other  users  of  the  LLC  participate  in  an 
educational  experience  that  leverages  the  unique  capabilities  of  blended  learning  systems.  For 
example,  instructors  must  deliver  curriculum  materials  by  posting  them  in  the  LLC. 

Although  curriculum  developers  and  the  technical  staff  are  responsible  for  creating  and 
migrating  course  content,  instructor  updates  posted  in  the  LLC  ensure  that  content  is  (a) 
accessible  to  all  users  of  the  LLC  (with  the  appropriate  permissions);  and  (b)  relieved  of  the 
limitations  associated  with  a  lengthy  curriculum  development  and  review  process.  Other  means 
for  delivering  curriculum  updates,  such  as  print  or  email,  prevent  the  achievement  of  some  key 
goals  of  the  system,  such  as  cost-effectiveness  and  uniform  access.  Instructors  may  customize 
course  content  based  on  student  feedback  by  leveraging  the  easy  access  to  Army  knowledge 
management  and  other  resources  that  the  LLC  provides.  Instructors  also  should  use  the  system  to 
perform  course  administrative  duties,  which  preserves  classroom  time  for  actual  learning 
(Bourne,  1998).  Instructors  also  may  leverage  system  capabilities  to  evaluate  student  progress 
and  report  grades  to  school  administrators.  Creative  use  of  the  LLC  components  to  evaluate 
student  progress  may  involve  such  activities  as  asynchronous  discussions  and  distributed 
presentations  and  collective  exercises  (Bonk  &  Cummings,  1998).  Finally,  early  adopters  of  the 
LLC  components  should  lead  the  development  of  SOPs  for  leveraging  the  system  capabilities 
(e.g.,  collaborative  capabilities  in  Bb)  and  mentor  late  adopters  on  application  functionalities 
to  enhance  course  instruction.  As  with  curriculum  developers,  late-adopting  instructors  may  be 
those  who  initially  resist  system  use  or  who  are  new  to  the  faculty.  Such  knowledge  management 
practices  would  ensure  that  best  practice  is  internalized  and  disseminated. 

CBT/WBT  courseware  production  team.  The  CBT/WBT  courseware  production  team, 
when  present,  must  ensure  that  CBT/WBT  courseware  delivered  via  the  LLC  is  reliable, 
instructionally  sound,  efficiently  delivered,  and  cost-effective.  To  do  this,  project  teams  must 
be  maintained  to  perform  CBT/WBT  analysis,  design,  development,  implementation, 
maintenance,  and  validation.  The  maintenance  of  these  project  teams  ensures  that  specialists 
are  allocated  to  each  component  of  the  development  process.  That  is,  instructional 
designers/analysts  are  assigned  to  designing  and  validating  course  content  as  well  as  overseeing 
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the  course  generation  process.  Programmers  ensure  the  effective  implementation  and  technical 
support  of  the  courseware  and  media  specialists  enable  engaging  multimedia  presentation  of 
course  content.  The  courseware  production  team  also  provides  contractual,  technical,  and 
educational/quality  oversight  of  contractor-developed  CBTAVBT.  Where  the  demand  for 
distance-learning  courseware  is  too  great  for  the  in-house  team,  it  nevertheless  must  assist  in 
maintaining  the  quality  and  efficiency  of  the  development  process  through  consultation  and 
support.  Finally,  the  courseware  production  team  must  maintain  a  database  of  CBTAVBT 
technologies,  capabilities,  and  techniques  in  order  to  leverage  the  latest  knowledge  and  best 
practice  in  multimedia  instruction.  The  presence  of  such  a  database  ensures  that  all  members  of 
the  team  have  access  to  the  knowledge  they  need  to  make  the  courseware  as  effective  as 
possible. 

Leadership.  The  LLC  leadership  is  key  to  ensuring  that  all  stakeholders  are  maximally 
engaged  and  that  through  their  coordinated  activity  the  initiative  reaches  its  goals.  To  facilitate 
the  coordinated  activity  of  the  other  LLC  personnel,  the  leadership  must  provide  the  vision  that 
serves  as  a  shared  objective  for  all  involved.  The  vision  should  be  based  on  a  user  and 
stakeholder  needs  assessment  that  maximally  aligns  the  goals  of  the  initiative  with  the  interests 
and  needs  of  the  people  involved  and  with  the  Army’s  organizational  mission  (Cianciolo  et  al., 
2006;  DTIC,  2000;  Thach,  1993).  The  leadership  must  then  communicate  the  vision  and 
market  the  LLC  concept  to  stakeholders  (LLC  Master  Plan,  2005;  Willis,  2000).  Stakeholders 
include  curriculum  developers,  instructors,  courseware  developers,  technical  staff,  students,  and 
the  operational  Army  because  each  of  these  individuals  are  affected  by  the  introduction  of  a  new 
learning  initiative  and  their  buy-in  and  personal  investment  are  critical  determinants  of  its 
success.  To  ensure  the  effective  functioning  of  the  LLC,  the  leadership  must  procure  resources 
to  maintain/update  the  LLC,  oversee  operations,  prioritize  limited  resources  across  LLC 
functions,  and  initiate/organize  the  development  of  SOPs  for  leveraging  the  system 
capabilities.  Although  SOPs  must  ultimately  come  from  the  personnel  who  use  the  LLC 
components,  leadership  is  necessary  to  guarantee  that  the  task  of  SOP  development  is  taken  on 
and  that  the  key  individuals  are  involved  who  can  facilitate  the  process  and  ensure  SOP  adoption. 

Output 


24/7  uniform  access.  Explicit  throughout  the  LLC  Master  Plan  (March,  2005)  is  the 
vision  that  the  teaching  and  learning  environment  will  be  improved  via  integrating  resident  and 
distance-based  training  and  education  into  a  blended,  seamless  process  that  enables  standardized 
instruction  anytime,  anywhere.  As  stated  in  the  planning  documentation,  the  main  output  of  the 
LLC  then  is  “relevant,  standardized  training  and  education,  on  demand,  to  the  right  Soldier, 
leader,  or  unit,  at  the  teachable  moment,  regardless  of  location”  (p.  1).  LLC  output  may  be 
conceptualized  both  as  access  to  the  system  (i.e.,  the  system  truly  is  available  24/7  to  all 
members  of  a  diverse  user  population)  and  actual  use  of  the  system  by  students  and  other  users. 
Use  of  an  accessible  system  is  necessary  to  achieve  the  intended  educational  and  organizational 
impact  of  the  initiative,  and  to  link  the  activities  of  personnel  to  these  outcomes. 

Computer-/Web-based  courseware  for  conducting  proponent  courses.  Unique  to  the  Fort 
Leonard  Wood  LLC  is  the  presence  of  a  CBT/WBT  courseware  production  team,  which 
comprises  instructional  designers,  programmers,  and  media  specialists  who  collectively  perform 
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CBTAVBT  analysis,  design,  development,  implementation,  maintenance,  and  validation  for 
instructors.  The  output  of  this  team  should  be  courseware  that  is  readily  available  to  meet 
course  needs.  That  is,  the  courseware  should  be  reliably  playable  in  Distance  Learning  XXI 
classrooms,  digital  training  facilities,  and  any  other  location  by  which  distance  education  is 
accessed.  If  the  courseware  is  not  accessible,  it  cannot  achieve  enhanced  instruction  and  learning, 
no  matter  how  educationally  sound  and  engaging  it  is.  The  courseware  also  should  feature  the 
same  range  in  content  that  is  found  in  the  courses  that  are  delivered  via  distance  education. 

Outcomes  (Short-,  Intermediate-,  and  Long-term) 

LLCs  are  expected  to  have  a  wide  set  of  outcomes,  ranging  from  short-term  impact  on  the 
learning  and  education  process,  to  long-term  impact  on  organizational  culture.  Intermediate-term 
impact  on  individual  learners  is  believed  to  link  short-  and  long-term  outcomes  as  more  learners 
use  the  LLCs  more  often  to  achieve  lifelong  learning.  Each  outcome  component  of  the  LLC  logic 
model  described  below  was  defined  and  justified  according  to  (1)  explicitly  stated  goals  for  the 
LLC  initiative;  (2)  implicitly  recognized  changes  that  the  LLC  could  produce  in  learning  and 
perfonnance;  and  (3)  theoretical  outcomes  expected  from  blended  learning  environments. 

Short-term  Outcome  -  Improved  teaching  and  learning  environment.  The  LLC  planning 
documentation  (LLC  Operational  Requirements ,  December  2005)  states  that  one  high-level  goal 
of  the  initiative  is  an  “improved  teaching  and  learning  environment.”  The  temptation  to  define 
“improved  teaching  and  learning  environment”  in  terms  of  the  advanced  technological 
capabilities  that  constitute  LLCs  is  reflected  in  some  of  the  initial  metrics  drafted  by  LLC 
stakeholders,  which  target  largely  the  technical  functioning  of  the  initiatives.  However,  defining 
improvements  in  this  way  confounds  technology  with  its  intended  educational  effect.  This 
precludes  the  specification  of  the  link  between  technology  and  intermediate-term  LLC  outcomes, 
such  as  improved  student  performance,  enhanced  readiness,  and  the  like.  Identifying  how  LLCs 
achieve  improvements  to  the  teaching  and  learning  environment  must  answer  the  question  “What 
necessary  changes  to  teaching  and  learning  can  be  brought  about  by  the  implementation  of  a 
blended  learning  environment?”  Metrics  associated  with  this  outcome  must  capture  whether  the 
changes  are  indeed  occurring. 

Discussions  with  LLC  directors  and  other  stakeholders  revealed  that  the  LLCs  could 
enable  several  improvements  to  the  teaching  and  learning  environment.  Chief  among  these 
improvements  is  enhanced  relevance  of  training  and  educational  content,  that  is,  the 
correspondence  of  training  and  educational  content  to  the  demands  of  the  operational 
environment.  LLCs  are  thought  to  enhance  relevance  primarily  in  two  ways.  First,  LLCs  connect 
the  institutional  Army  to  the  operational  Army  through  centralized  access  to  the  Battle 
Command  Knowledge  System  Leader  Network  and  numerous  other  online  discussion  forums. 
This  connection  is  believed  to  reduce  the  time  required  to  incorporate  operational  lessons  learned 
into  course  content  because  input  comes  directly  from  the  field  rather  than  through  a  lengthy 
institutional  review  and  vetting  process.  Second,  the  online  presence  of  LLCs  eliminates  the 
requirement  to  replicate  and  ship  training  and  educational  materials  to  students  or  other  learners 
who  are  not  located  at  the  schoolhouse.  Replication  and  shipping  can  add  a  significant  time  delay 
to  distributing  instructional  materials  (sometimes  up  to  a  year),  expanding  the  gap  between 
learning  content  and  the  reality  of  the  operational  environment.  Increased  correspondence 


12 


between  training  and  educational  content  and  the  operational  environment  may  also  be  achieved 
through  the  timing  of  instructional  delivery.  In  some  contexts,  just-in-time  instruction  may  be 
thought  of  as  more  relevant  than  “just-in-case”  instruction,  although  formal  education  must  strike 
a  balance  between  these  endeavors. 

Discussions  with  the  director  and  staff  of  the  Fort  Leonard  Wood  LLC  revealed  that  their 
LLC  specifically  addresses  the  limited  expertise  instructors  may  have  in  producing  courseware 
for  computer-based  training  (CBT)  and/or  web-based  training  (WBT)  for  distance  learning.  The 
Fort  Leonard  Wood  LLC  is  unique  among  the  LLCs  in  having  an  in-house  courseware 
production  team  that  comprises  instructional  designers,  programmers,  and  media  specialists  who 
collectively  perform  CBT/WBT  analysis,  design,  development,  implementation,  maintenance, 
and  validation  for  instructors.  This  courseware  development  team  improves  the  teaching  and 
learning  environment  by  enabling  the  delivery  of  more  advanced  CBT/WBT  courseware  for 
distributed/distance  learning  that  is  educationally  sound  and  compliant  with  Army  training 
regulations.  Specifically,  educationally  sound  courseware  should  meet  rigorous,  independently 
derived  criteria  for  content,  instructional  activities,  performance  assessment  and  feedback,  and 
usability  (Hays,  Stout,  and  Ryan-Jones,  2005).  Therefore,  the  delivery  of  CBT/WBT  courseware 
produced  by  specialists  may  enhance  the  teaching  and  learning  environment  by  obviating 
individual  differences  in  instructor  capability  to  produce  engaging,  effective,  and  usable 
courseware. 

Review  of  the  scholarly  literature  indicated  that  there  are  several  ways  by  which 
introducing  blended  learning  to  the  instructional  process  may  improve  both  teaching  and 
learning.  The  most  commonly  cited  improvement  is  the  emergence  of  a  learning  experience  in 
which  the  instructor  acts  as  a  facilitator  of  adult  learning,  fostering  knowledge-seeking  and 
knowledge-generation  in  his  or  her  students  through  wide-ranging  access  to  resources,  people, 
and  information.  As  stated  in  Firdyiwek  (1999),  “the  pedagogy  of  online  instruction  is  based  on 
the  effective  use  of  electronic  learning  environments  for  the  development  of  cognitive  skills 
through  access  to  information,  interactivity  with  tools,  and  communication”  (p.  29).  There  is 
general  agreement  in  the  literature  (e.g.,  Abell,  2003;  American  Distance  Education  Consortium, 
2003;  Bonk  &  Cummings,  1998;  Bonk  &  Reynolds,  1997;  Chickering  &  Gamson,  1987; 
Herrington,  Herrington,  Oliver,  Stoney,  &  Willis,  2001;  Mason,  1991;  McLoughlin  &  Oliver, 

1 999;  University  of  Illinois,  Urbana-Champaign,  1 999)  that  the  following  instructor  behaviors 
reflect  facilitation  of  adult  learning  and  effective  use  of  educational  technology: 

•  Provides  clear  guidance  on  expectations  and  evaluation  criteria 

•  Provides  positive  and  informative  feedback 

•  Provides  challenges  to  students  without  overwhelming  them 

•  Assigns  group  tasks  and  roles  within  group  tasks 

•  Involves  students  in  generating  course  material 

•  Facilitates  analysis  and  reflection  through  questioning 

•  Encourages  students  to  use  resources  (including  other  experts)  outside  the  classroom  for 

research  and  learning 

These  behaviors  are  applicable  to  any  classroom,  traditional  or  otherwise  (e.g.,  Abell, 
2000),  but  they  are  especially  important  in  distributed  learning  situations  in  which  students  lack 
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the  social  cues  and  interpersonal  connections  they  receive  in  face-to-face  settings  (Hiltz,  1998; 
Kreijns,  Kirschner,  Jochems,  2003).  Failure  to  leverage  the  technological  capabilities  present  in 
blended  learning  solutions  may  reduce  the  educational  effectiveness  of  instruction  relative  to 
traditional  classrooms  by  creating  confusion,  frustration  with  the  learning  experience,  and  social 
distance. 

For  students  learning  synchronously  in  a  classroom  setting  (traditional  or  virtual), 
enhanced  instructor  efficiency  is  another  improvement  to  the  teaching  and  learning 
environment  enabled  by  blended  learning  solutions.  A  wide  range  of  scholars  and  practitioners 
agree  [e.g.,  Clark,  1994;  Defense  Technical  Information  Center  (DTIC),  2000;  Ehnnann,  1994; 
Firdyiwek,  1999;  Hannafm  &  Land,  1997;  Owston,  1997;  Willis,  2000]  that  the  long-range 
impact  of  technology  on  learning  is  a  function  of  the  degree  to  which  the  technology  is  leveraged 
to  support  change  in  the  learning  process.  Such  change  is  partly  enabled  when  instructors  use  the 
administrative  functions  of  instructional  technologies  to  protect  valuable  classroom  learning  time 
from  administrative  interruptions  (Bourne,  1998).  Instructors  enhance  classroom  efficiency  by 
using  asynchronous  collaborative  tools  to  prepare  students  for  class  discussions,  to  make 
announcements,  to  collect  homework,  to  administer  exams,  and  so  on.  The  “extra”  available 
class  time  may  then  be  used  for  more  in-depth  exploration  or  processing  of  ideas  through 
discussion  or  practical  group  exercises. 

The  presence  of  a  learning  community  is  considered  a  key  hallmark  of  an  effectively 
facilitated  adult  classroom,  especially  one  supported  by  blended  learning  technologies  that 
enable  learners  to  connect  and  share  easily  anytime,  from  anywhere.  In  the  context  of  blended 
learning,  learning  communities  may  be  defined  as  networks  of  learners  who  are  tied  to  one 
another  through  the  exchange  of  intellectual,  practical,  and/or  social  support  (Haythomthwaite, 
2002;  Moller,  1998;  Wellman,  Carrington,  &  Hall,  1988).  Learning  communities  characterized 
by  strong  ties  between  a  small  number  of  individuals  enhance  the  teaching  and  learning 
environment  by  (a)  providing  social  support  critical  to  retention  and  educational  success, 
especially  for  distance  learners  (Haythomthwaite,  2005;  Kreijns  et  al.,  2003;  Sanders  & 

Burnside,  2001);  and  (b)  transforming  the  learning  environment  into  a  vehicle  for  organizational 
socialization  through  the  growth  of  shared  experience  and  values  (Chao,  O’Leary-Kelly,  Wolf, 
Klein,  &  Gardner,  1994;  McMillan  &  Chavis,  1986).  Large  learning  communities  characterized 
by  a  greater  number  of  weak,  diffuse  ties— in  effect  communities  of  practice  (Wenger, 

McDermott,  &  Snyder,  2002)— facilitate  information  sharing  among  diverse  people  and  new 
knowledge  development  (Haythomthwaite,  2002;  Wellman  &  Gulia,  1999).  Learning 
communities  are  more  easily  formed  when  learners  are  within  close  proximity  of  one  another, 
but  ties  that  enable  the  exchange  of  support  may  be  formed  and  maintained  even  at  a  distance 
(Sanders,  2002;  Wellman  &  Gulia,  1999). 

Intermediate-term  outcome  -  Improved  student  performance.  The  LLC  Master  Plan 
(March,  2005)  states  that  LLCs  will  “improve  Soldier  performance  through  resident  training  and 
distance  learning.”  Referring  to  the  2004  fiscal  year  training  guidance  given  by  the  Commanding 
General  of  the  U.S.  Army  Training  and  Doctrine  Command  (TRADOC),  the  master  plan  implies 
that  improved  student  perfonnance  will  feature  personal  responsibility  for  learning  and  enhanced 
skill  development.  In  theory,  improved  student  performance  results  from  the  effective  use  of 
educational  technology  to  revolutionize  instruction  and  enhance  the  meaningfulness  of  course 


14 


content.  Importantly,  student  performance  in  this  research  primarily  is  conceptualized  in  terms  of 
general  learner  competencies  that  may  be  improved  through  the  effective  use  of  educational 
technology.  Conceptualizing  student  performance  in  this  way  makes  the  logic  model  more 
generalizable  across  LLCs  while  capturing  aspects  of  performance  that  are  most  closely  linked  to 
the  changes  in  instruction  brought  about  by  LLCs. 

The  contemporary  operating  environment  has  placed  greater  demand  on  Soldiers  and 
leaders  to  have  a  diverse,  robust  skill  set  that  includes  both  individual  and  team-based 
capabilities.  Diverse  skill  sets  are  required  so  that  Army  personnel  can  effectively  carry  out  full- 
spectrum  operations  with  daily  changing  mission  requirements.  Soldiers  and  leaders  also  must 
acquire  and  retain  robust  behavioral  repertoires  that  transfer  to  a  variety  of  performance 
environments,  such  as  urban  versus  open  terrain.  They  have  precious  few  opportunities  to  train 
as  collectives  in  the  unit  setting  and  so  must  rely  on  institutional  training  to  develop  the 
individual  and  team-based  competences  necessary  for  collective  performance.  Enhanced  skill 
development  therefore  comprises  a  greater  range  in  skill  sets  acquired  by  Soldiers  and  leaders, 
greater  transfer  of  training  to  diverse  performance  environments,  and  improved  collective  task 
execution.  Skill  development  may  be  enhanced  via  blended  learning  through  more  field-relevant 
instructional  content,  more  frequent  and  varied  training  opportunities,  more  productive  learning 
time  due  to  reduced  administration  in  the  classroom,  and  more  frequent  opportunities  to 
participate  in  group  learning  activities  (e.g.,  Abell,  n.d.;  Bonk  &  Reynolds,  1997;  Bourne,  1998). 

Enhanced  learner  independence  and  responsibility  follows  from  effective  instructor 
facilitation  in  the  blended  learning  environment  (McLoughlin  &  Oliver,  1999).  Although 
generally  not  studied  as  its  own  subject,  it  may  be  conceptualized  as  one’s  sense  of  ownership  of 
one’s  professional  development  and  learning  process,  indicated  by  establishing  learning  goals, 
engaging  in  independent  research,  preparing  for  learning  opportunities  (e.g.,  class  discussions), 
and  maintaining  regular  contact  with  learning  resources.  Although  the  development  of  learner 
independence  and  responsibility  is  an  important  outcome  of  any  learning  environment,  the 
blended  learning  environment  is  believed  to  enhance  these  characteristics  by  making  it  easier  for 
instructors  to  promote  increased  student  participation  in  facilitated  group  activities  (e.g., 
asynchronous  conferencing),  learning  self-management  (e.g.,  database  and/or  portfolio 
generation),  and  independent  research  (e.g.,  internet  searches  and/or  participation  in  online 
communities  of  practice)  (McLoughlin  &  Oliver,  1 999). 

The  behaviors  reflecting  learner  independence  and  responsibility  are  tightly  coupled  with 
other  learner  characteristics  that  also  are  influenced  by  instructor  effectiveness,  namely 
enhanced  learning  self-efficacy  (Abell,  2003;  Gibson,  1996)  and  enhanced  motivation  (Abell, 
2000;  Cornell  &  Martin,  1997).  Learning  self-efficacy  within  the  context  of  blended  learning  is 
reflected  in  students’  perception  of  their  ability  to  lead  their  own  learning  process  and  to  achieve 
their  learning  goals  using  the  information,  people,  and  other  resources  available  to  them  through 
online  content  and  collaborative  tools.  One’s  learning  self-efficacy  may  be  negatively  affected 
by  educational  technology  if  the  technology  is  not  usable  or  otherwise  introduces  difficulty  to  the 
learning  process  due,  for  example,  to  reduced  performance  feedback  or  social  interaction  (Abell, 
2003;  Gibson,  1996;  Haythomthwaite,  2005;  Kreijns  et  ah,  2003).  In  addition  to  instructor 
effectiveness,  the  job-related  applicability  of  the  learning  content  is  a  well-recognized  motivator 
of  adult  learners  (Abell,  2000;  O’Donnell,  2005).  To  the  extent  that  the  blended  learning 
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environment  is  leveraged  to  enhance  the  relevance  of  course  content  and  the  performance- 
environment  representativeness  of  learning  activities  (e.g.,  through  simulations  and  collaborative 
activities),  learner  motivation  can  be  expected  to  increase  (Cornell  &  Martin,  1 997). 

Enhanced  higher-order  thinking  and  enhanced  reflective  capability  are  long- 
recognized  student-performance  goals  of  any  educational  initiative  (Dewey,  1933;  Garrison, 

1991 ;  Lipman,  1987;  Nickerson,  1989).  The  introduction  of  blended  learning  technologies  into 
the  learning  process  does  not  obviate  these  goals,  but  may  enhance  their  achievement  by 
increasing  the  access  students  have  to  diverse  perspectives  and  cognitive  resources  (Grotzer  & 
Perkins,  2000;  Haythomthwaite,  2002;  McLoughlin  &  Oliver,  1999;  Paul,  1987)  and  the  time 
students  have  to  engage  in  guided  critical  thinking,  problem  solving,  and  reflective  practice  in 
the  classroom  (Bourne,  1998;  Grotzer  &  Perkins,  2000).  Enhanced  higher-order  thinking  and 
reflective  capability  may  be  captured  through  special  purpose  measures  that  reflect  critical 
thinking  and  problem  solving  skills  or  that  exercise  the  process  of  reflection. 

Intermediate-term  outcome  -  Improved  individual  and  unit  performance  in  the  field. 
Highlighted  in  the  LLC  Master  Plan  (March,  2005)  is  the  importance  of  supporting  TRADOC’s 
imperative  to  “provide  Soldiers  and  leaders  who  can  immediately  contribute  to  unit  readiness  on 
the  first  day  they  arrive  in  their  unit”  (p.  5).  Similarly,  the  Army  must  deploy  cohesive  units  that 
can  immediately  contribute  to  mission  accomplishment  the  day  they  arrive  in  theater.  Rapid 
readiness  is  dependent  in  part  on  the  retention  and  adaptation  of  skills  and  competencies  as  well 
as  the  swift  acquisition  of  new  ones.  A  key  aspect  of  individual  and  unit  performance  in  the  field 
that  can  be  influenced  by  LLCs,  then,  is  just-in-time  competency.  Swift  skill  acquisition  is 
enabled  by  anytime,  anywhere  access  to  one’s  proponent  schoolhouse,  a  defining  characteristic 
of  the  LLC.  The  retention  and  adaptation  of  skills  and  competencies  that  support  just-in-time 
competency  stems  in  part  from  the  enhanced  cognitive  ability  reflected  in  improved  student 
performance,  as  described  above,  and  in  part  from  the  adoption  of  a  lifelong  learning  orientation, 
as  described  below. 

Intermediate-term  outcome  -  Adoption  of  lifelong  learning  orientation.  The  adoption  of  a 
lifelong  learning  orientation  is  a  signal  outcome  of  the  LLC  initiative  and  a  critical  determinant 
of  its  perpetuation.  The  LLC  Master  Plan  (March,  2005)  states:  “Soldiers  must  perceive  learning 
as  a  continuous  process  that  must  be  refreshed  throughout  their  career.”  Lifelong  learning 
orientation  might  therefore  be  conceptualized  as  one’s  distal  motivation  to  engage  in 
opportunities  to  learn.  Distal  motivation  is  one’s  intent  to  engage  in  a  particular  task,  is 
reflected  in  the  goals  one  sets  for  task  performance,  and  determines  how  one  allocates  effort 
across  the  range  of  possible  tasks  (Kanfer  &  Ackerman,  1989).  A  Soldier  or  leader  who  intends 
to  engage  in  continuous  learning  will  set  goals  for  participation  in  learning  opportunities  and 
prioritize  their  efforts  to  target  lifelong  learning.  A  person’s  distal  motivation  is  a  function  of 
three  subjective  assessments:  (1)  the  perceived  utility  of  performance  (i.e.,  the  desirability/payoff 
of  the  chosen  goal);  (2)  the  perceived  utility  of  effort  (i.e.,  one’s  intrinsic  need  to  achieve);  and 
(3)  the  perceived  relation  between  one’s  effort  and  performance  (e.g.,  as  indicated  in 
performance  feedback  with  regard  to  goal  achievement)  (Kanfer,  1987).  Put  another  way,  people 
will  choose  to  engage  in  opportunities  to  learn  if  they  feel  that  lifelong  learning  has  certain 
benefits,  that  goal  achievement  is  worthy  of  effort  in  and  of  itself,  and  that  engagement  in 
specific  lifelong  learning  opportunities  will  result  in  goal  achievement  and  payoff. 
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The  relation  between  a  blended  learning  initiative  and  intent  to  engage  in  opportunities  to 
learn  is  complicated.  A  blended  learning  initiative  may  not  be  as  dominant  as  some 
organizational  characteristics  (e.g.,  promotion  policy)  in  driving  the  perceived  payoff  of  lifelong 
learning.  In  addition,  external  factors  may  play  a  larger  role  in  determining  people’s  intrinsic 
need  for  achievement  than  characteristics  of  the  learning  environment.  However,  the 
effectiveness  of  the  LLCs  may  have  a  strong  impact  on  Solders’  and  leaders’  perception  that 
engaging  in  lifelong  learning  will  help  them  to  achieve  their  professional  goals,  as  opposed  to 
some  other  activity.  The  integration  of  education  and  training  initiatives  on  the  one  hand  and 
personnel  initiatives  on  the  other  may  be  the  most  effective  way  to  facilitate  the  development  of 
lifelong  learning  orientation  (Moran,  2006). 

Intermediate-term  outcome  -  Enhanced  mission  readiness.  The  LLC  Operational 
Requirements  (December,  2005)  state  that  the  LLCs  will  provide  “improved  training  and 
education  support”  to  the  Field  Commander  and  deployed  Army.  Implicit  in  this  statement  is  the 
goal  of  achieving  enhanced  mission  readiness,  an  objective  of  particular  interest  to  the  Army. 
Readiness  maybe  conceptualized  in  two  ways:  (1)  as  unit  status  [i.e.,  U.S.  Department  of  the 
Army  (DA),  2006];  and  (2)  as  individual  and  team  characteristics  that  affect  unit  performance 
(e.g.,  McGonigle,  Casper,  Meiman,  Cronin,  Cronin,  &  Harris,  2005).  LLCs  may  have  an  impact 
on  both  of  these  aspects  of  readiness  over  both  the  intermediate-  and  long-term. 

Unit  readiness,  as  captured  by  unit  status  reports,  is  a  function  of  personnel  numbers, 
equipment  availability,  equipment  readiness,  and  training  status  (DA,  2006).  LLCs,  by  enabling 
anytime,  anywhere  access  to  standardized  training  materials,  may  be  expected  to  enhance  unit 
status  reporting— specifically  the  reason  codes  associated  with  sub-optimal  personnel  and 
training  components  of  unit  readiness.  Personnel  readiness  is  determined  by  comparing  the 
available  strength,  the  available  military  occupational  specialty  qualified  (MOSQ)  strength,  and 
the  available  senior  grade  strength  to  the  required  unit  strength.  Of  particular  relevance  to  LLCs 
that  provide  MOS-qualification  training  is  the  available  MOSQ  personnel  percentage  component 
of  personnel  readiness.  The  MOSQ  personnel  percentage  is  the  ratio  of  available  MOSQ 
personnel  to  required  MOSQ  personnel.  It  is  reduced  when  Soldiers  cannot  attend  MOS- 
qualification  training  when  they  need  it.  A  unit’s  training  status  is  determined  jointly  by  the 
commander’s  estimate  of  the  proportion  of  the  mission-essential  task  list  (METL)  that  has  been 
trained  to  standard  and  the  number  of  training  days  required  to  reach  standard  on  METL  core 
tasks.  Training  readiness  estimates  may  be  influenced  by  a  number  of  factors,  including 
personnel  absences  during  training  exercises  because  they  do  not  have  the  required  MOS- 
qualification  training  or  absences  due  to  attending  MOS-qualification  training  at  the 
schoolhouse.  Reserve  units  experience  great  difficulty  reaching  optimal  training  status  for  these 
reasons  (Sortor,  Lippiatt,  Polich,  &  Crowley,  1994).  By  making  MOS-qualification  training 
available  anytime  and  anywhere,  LLCs  may  enhance  unit  status  reporting  by  reducing  the 
number  of  Soldiers  who  are  not  qualified  for  their  duty  positions  or  who  must  travel  to 
participate  in  qualification  training. 

LLCs  also  should  reduce  the  time  to  optimal  training  status  (as  reflected  in  unit  status 
reports)  by  allowing  more  people  to  complete  MOS-qualification  training  in  a  timely  fashion. 
This  is  because  anytime,  anywhere  training  enabled  by  the  LLC  blended  learning  environment  is 
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not  limited  by  the  number  of  seats  available  in  the  schoolhouse  (see  also  Freeman,  2003).  Those 
who  need  MOS-qualification  training  can  receive  it  just  in  time,  rather  than  waiting  until  the 
schoolhouse  has  the  space  to  seat  them.  Having  MOS-qualified  (MOSQ)  personnel  available  for 
unit  training  should  reduce  the  number  of  days  required  to  reach  standard  on  METL  core  tasks 
by  improving  training  quality  (Sortor  et  al.,  1994). 

Along  these  lines,  LLCs  may  enhance  MOSQ  training  status,  that  is,  the  number  of 
individuals  receiving  MOS-qualification  training  relative  to  those  who  need  it.  For  example, 
there  are  a  great  many  more  individuals  requiring  MOSQ  and  leader  training  in  the  reserve 
component  than  who  are  able  to  receive  it  through  residential  schooling  (Winkler,  Shanley, 
Crowley,  Madison,  Green,  Polich,  et  ah,  1996).  LLCs  may  affect  the  proportion  of  individuals 
both  requiring  MOS-qualification  training  and  receiving  it  by  making  “virtual”  training  seats 
available  to  anyone.  The  use  of  training  seats  available  by  the  reserve  component  also  is 
inefficient,  but  this  is  likely  due  to  factors  outside  of  the  sphere  of  influence  of  LLCs  (Brown, 
2002). 


Arguments  have  been  made  against  the  adequacy  of  unit  status  reporting  (e.g.,  James, 
2004;  Moore,  Stockfisch,  Goldberg,  Holroyd,  Hildebrandt,  1991;  Sortor  et  ah,  1994;  Winkler  et 
ah,  1996),  in  part  because  unit  status  reporting  is  subjective  and  because  it  does  not  capture  all  of 
the  determinants  of  mission  success  (e.g.,  leadership  quality,  troop  morale,  and  proportion  of 
low-density/high-demand  DMOSQ  personnel).  In  addition,  unit  status  ratings  are  very  difficult 
to  change  for  a  variety  of  reasons  (Moore  et  ah,  1991).  To  the  extent  that  there  is  reduced 
variance  in  unit  status  reports  due  to  subjectivity  or  reduced  external  validity  due  to  lack  of 
content  validity,  the  utility  of  unit  status  reports  as  indicators  of  readiness  (and  LLC  impact  on 
readiness)  will  be  limited. 

Behavioral  theories  of  readiness  partially  address  the  challenges  associated  with  unit 
status  measures  of  readiness.  The  advantage  of  behavioral  theories  is  that  they  lend  themselves 
well  to  using  psychometrically  sound  measures  of  readiness.  The  disadvantage  of  such  theories  is 
that  the  link  between  the  identified  readiness  components  and  actual  mission  performance  is  not 
fully  known1.  Components  of  readiness  in  the  psychological  literature  include  unit  cohesion, 
physical  fitness,  technical  competence/job  performance,  organizational  citizenship  behavior 
(e.g.,  helping  behaviors,  punctuality,  etc.),  preparedness  to  deploy,  and  organizational 
commitment/retention  (McGonigle  et  al.,  2005).  These  components  are  in  turn  influenced  by 
skill  development,  job  satisfaction,  family  adaptation,  perceived  organizational  support,  and 
self/collective  efficacy  (McGonigle  et  al.,  2005). 

LLCs  may  improve  readiness  by  enhancing  (affective)  organizational  commitment. 
Affective  organizational  commitment  is  distinguished  from  other  forms  of  commitment 
(normative  and  continuance)  by  its  emotional  characteristic  (i.e.,  organizational  members  stay 
because  they  want  to)  and  its  positive  relation  to  organizational  measures  of  success  (e.g., 
efficiency;  McGonigle  et  al.,  2005).  LLCs  may  influence  affective  organizational  commitment 
by  buttressing  users’  perceptions  of  organizational  support.  As  described  in  Chao  (2006),  any 
interaction  an  individual  has  with  the  organization— whether  through  other  members  of  the 
organization,  through  organizational  initiatives,  or  some  other  means— acts  as  an  agent  of 

1  Note  this  same  criticism  has  been  leveled  against  unit  status  reports  (e.g.,  Junor,  2005;  Moore  et  al.,  1991). 
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organizational  socialization.  To  the  extent  that  LLCs  convey  to  users  that  the  Army  is  doing  its 
best  to  support  Soldier  and  leader  education  (i.e.,  by  meeting  learners’  needs),  affective 
commitment  maybe  increased  (Bumam,  Meredith,  Sherboume,  Valdez,  &  Vemez,  1992;  Gibson 
&  Tremble,  2006).  Conversely,  if  the  LLC  fails  to  meet  learners’  needs  or  presents  a  learning 
environment  characterized  by  poor  usability  and/or  significant  technical  difficulty,  affective 
commitment  may  be  decreased  as  learners  become  cynical  about  the  Army’s  ability  and 
motivation  to  support  their  professional  development  (Bumam  et  al.,  1992;  Gibson  &  Tremble, 
2006). 


Along  these  lines,  LLCs  may  improve  readiness  through  enhanced  socialization  in 
organizational  goals  and  values.  In  a  five-year  study  of  engineering,  management,  and  other 
professionals,  Chao  et  al.  (1994)  demonstrated  that  study  participants  who  changed  organizations 
were  significantly  less  socialized  in  organizational  goals  and  values  prior  to  leaving  their 
employers,  suggesting  a  link  between  this  form  of  socialization  and  organizational  commitment. 
Moreover,  socialization  in  organizational  goals  and  values  had  the  strongest  relation  among  six 
socialization  dimensions  (people,  politics,  history,  job  proficiency,  language,  and  organizational 
goals  and  values)  to  career  involvement  and  job  satisfaction  and  accounted  for  significant 
variance  in  these  variables  above  and  beyond  job  tenure  and  organizational  tenure.  Through 
effective  instructor  facilitation  of  distance  learning  and  the  development  of  learning 
communities,  LLCs  contribute  to  this  socialization  process.  The  findings  of  Sanders  (2002)  and 
Wellman  and  Gulia  (1999)  indicate  that  distance  between  learners  does  not  necessarily  hamper 
the  development  of  social  bonds  and  learning  communities.  Therefore,  to  the  extent  that 
instructor-student,  student-student,  and  student-curriculum  interaction  using  the  LLCs  reinforces 
shared  organizational  goals  and  values,  increases  may  be  observed  in  job  satisfaction, 
performance  (i.e.,  job-related  problems),  and  organizational  commitment  (see  also  Major, 
Kozlowski,  Chao,  &  Gardner,  1995). 

LLCs  also  may  affect  readiness  through  reductions  in  work-education-family  conflict, 
however  it  is  unknown  how  well  they  will  do  so.  Adams,  Jex,  and  Cunningham  (2006)  identified 
three  types  of  work-family  conflict:  (1)  time-based  (actual  and  perceived  time  available, 
including  separations  from  family);  (2)  strain-based  (work  strain  producing  family  strain);  and 
(3)  behavior-based  (e.g.,  role  conflict,  family  versus  student).  Although  anytime,  anywhere 
education  and  training  reduces  learner  travel  requirements,  the  length  of  time  away  from  family 
has  been  found  to  be  generally  unassociated  with  various  individual  readiness  factors,  such  as 
job-related  problems,  lost  duty  time,  absence  from  alerts  or  deployments,  and  organizational 
commitment  (Bumam  et  al.,  1992).  However,  conflicts  between  the  demands  of  work,  education, 
and  family  have  been  identified  as  determinants  of  drop  out  rates  from  distance  education  and 
reduced  learning  achievement  (e.g.,  Haythomthwaite,  2005;  Kreijns  et  al.,  2003;  Phelps  et  al., 
1991 ;  Sanders  &  Guyer,  2001).  Research  suggests  that  the  negative  impact  of  learning  at  a 
distance  may  be  alleviated  through  the  development  of  learning  communities,  perceptions  that 
the  learning  environment  represents  organizational  support  for  its  members,  and  perceptions  that 
the  time  spent  on  education  is  necessary  (Bumam  et  al.,  1992;  Gibson  &  Tremble,  2006; 
Haythomthwaite,  2005).  Anytime,  anywhere  education  enabled  by  LLCs  might  reduce  work- 
education-family  conflict;  it  must  not  increase  it. 
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Long-term  outcome  -  Culture  shift.  The  LLC  Master  Plan  (March  2005)  refers  to  the 
following  quote  from  the  Army  Chief  of  Staff  Army  Knowledge  Management  Strategic  Plan: 
“Becoming  a  knowledge  based  organization  involves  more  than  technologies  -  it  requires  deep 
cultural  shifts  -  from  traditional  practices  to  collaboration,  teamwork,  and  innovation;  from 
information  hoarding  to  knowledge  sharing;  from  traditional  skills  to  Internet-Age 
competencies.”  This  reference  illustrates  the  primary  long-term  goal  of  the  lifelong  learning 
initiative,  facilitating  this  cultural  shift  through  access  to  advanced,  collaborative  technologies. 
The  means  by  which  LLCs  can  have  an  impact  on  cultural  shift  is  by  enhancing  collaboration 
orientation  and  fostering  internalization  of  anytime,  anywhere  learning  through  repeated 
individual  success  with  anytime,  anywhere  learning.  The  success  of  LLCs  in  accomplishing 
these  goals  depends  on  how  well  the  distributed  learning  environment  is  leveraged  to  create  the 
sense  of  a  broad  community  of  learners  of  which  everyone,  including  instructors,  is  an  important 
part. 


Sherry  and  Wilson  (1997)  argue  that  transformative  communication,  an  alternative  to 
instructor-dominated  instruction  that  is  enabled  by  learning  environments  with  internet  access, 
promotes  the  internalization  of  anytime,  anywhere  learning.  Students  may  find  they  have 
something  to  teach  the  instructor  about  the  technology  used  in  the  course  and  also  that  they  have 
easy  access  to  outside  experts  (e.g.,  through  websites  and  online  discussion  forums).  Rather  than 
the  opposite,  students  may  call  the  instructor’s  attention  to  these  valuable  learning  resources. 
Through  asynchronous  and  synchronous  distributed  collaboration,  activity  in  the  blended 
learning  environment  role-models  a  change  in  thinking  about  what  a  “classroom”  is— from 
conventional  notions  of  a  lecture  hall  or  desk  and  whiteboard  to  home  offices,  cafes,  and  even 
conversations  around  the  water  cooler.  To  the  extent  that  students  begin  to  see  just  about 
anything  as  an  opportunity  to  learn,  they  have  internalized  anytime,  anywhere  learning.  Along 
similar  lines,  blended  learning  environments— especially  those  that  connect  to  a  wide  community 
of  experts  and  peers— may  enhance  collaboration  orientation  by  expanding  learners’  conception 
of  the  learner  network  and  their  understanding  of  their  role  in  facilitating  the  collective 
knowledge  acquisition  of  this  network. 

Long-term  outcome  -  Enhanced  educational  cost-effectiveness.  Although  the 
establishment  of  a  blended  learning  environment  is  associated  with  significant  capital 
investment,  it  potentially  presents  a  more  cost-effective  training  and  education  solution  over  the 
long-tenn  due  to  reduced  variable  costs  such  as  printing  and  shipping  (Whalen  &  Wright,  1999). 
Yet,  due  to  the  significant  expense  of  technology-assisted  education  it  is  not  recommended  for 
the  purposes  of  saving  costs,  but  for  expanding  academic  outreach  (Willis,  2000).  However,  the 
increased  cost  associated  with  technology  procurement,  staffing,  and  housing  may  be  offset  by 
greater  outreach,  resulting  in  a  lower  cost  per  student  than  traditional  alternatives.  LLCs  can 
achieve  this  enhanced  cost-outreach  by  providing  access  to  a  greater  number  of  students  who 
may  participate  in  institutional  education  from  home  and  deployed  locations.  Increased  outreach 
also  would  enhance  throughput  effectiveness  relative  to  previous  conditions  as  more  people 
who  require  training  and  education  are  able  to  receive  it.  LLCs  situated  at  schoolhouses  that 
maintain  ranges,  equipment,  and  simulations  for  training  may  offset  costs  associated  with 
technology  procurement  by  reducing  the  range  equipment/supplies  requirements  to  conduct 
training.  Finally,  LLCs  with  in-house  CBT/WBT  courseware  production  teams,  such  as  the  Fort 
Leonard  Wood  LLC,  may  enhance  CBTAYBT  courseware  development  cost-effectiveness 
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relative  to  working  with  outside  contractors.  Such  cost  savings  were  demonstrated  in  Phelps  et 
al.  (1991). 


METRICS  AND  MEASURES  FOR  EVALUATING  LLCS 

Two-hundred  and  twenty-nine  metrics  and  associated  measures  were  developed  to  reflect 
the  above-described  components  of  the  LLC  logic  model.  A  complete  listing  of  these  metrics  and 
measures  is  featured  in  Appendix  A.  The  metric  chart  is  organized  according  to  the  logic  model 
structure  discussed  above,  with  metrics  reflecting  the  status  of  the  LLC  on  each  of  the  model 
components,  subcomponents,  and  elements  (e.g.,  Activities  -  Technical  Staff  -  Provide  technical 
support).  Metrics  were  designed  to  enable  sampling  of  users  or  courses,  rather  than  requiring 
assessment  of  an  LLC  in  its  entirety.  Because  the  LLCs  feature  multiple  levels  of  blended 
learning  (Graham,  2006)  metrics  in  the  chart  are  associated  with  a  description  of  the  type  of 
learning  environment  to  which  they  apply  as  well  as  their  corresponding  measure.  Specifying  the 
applicable  learning  environment  should  buttress  the  generalizability  of  the  chart  through  an 
examination  of  the  level  of  blending  and  of  the  learning  environments  used  in  a  particular  LLC. 
The  metric  chart  also  depicts  the  pilot  LLCs  to  which  each  metric  applies. 

The  key  design  requirements  for  the  metric  measures  were  feasibility  and  utility.  In  order 
to  be  used,  the  measures  must  not  require  a  great  deal  of  in-depth  system  analysis  or  complex 
data  analyses.  Yet,  the  measures  must  provide  useful  information  about  the  metrics  that  rule  out, 
to  the  extent  possible,  alternative  interpretations  of  metric  data.  Candidate  measures  featured  in 
the  metrics  and  measures  chart  in  Appendix  A  include  surveys,  interviews,  archival  data 
collection  (e.g.,  from  Army  personnel  databases  or  schoolhouse  financial  reports),  and  some  low- 
level  system  analysis. 

The  metrics  and  measures  presented  in  Appendix  A  represent  an  initial  attempt  to  specify 
the  assessment  criteria  for  evaluating  LLCs.  Some  modification  to  this  chart  may  be  necessary, 
given  that  not  all  metrics  and  measures  were  tested  in  the  present  investigation.  Ideally,  all  of  the 
metrics  in  the  chart  applicable  to  a  particular  LLC  would  be  used  in  an  evaluation.  This  may  not 
be  feasible  given  time  constraints  or  may  not  be  necessary  for  well  established  LLCs. 

APPLICATION  OF  THE  LOGIC  MODEL  AND  METRICS  TO  EVALUATING  THE 
FORT  LEAVENWORTH  LIFELONG  LEARNING  INITIATIVE 

The  LLC  logic  model  and  associated  metrics  were  used  to  conduct  a  formative  evaluation 
of  Fort  Leavenworth’s  LLC,  also  called  the  Fort  Leavenworth  Lifelong  Learning  Initiative  (LVN 
LLI).  The  complete  description  of  this  evaluation  is  presented  in  Appendix  B.  Data  were 
collected  during  the  months  of  September  and  October  2006  using  surveys  and  interviews, 
system  analysis,  and  retrieval  of  archival  financial  and  enrollment  data.  Data  collection  largely 
focused  on  the  application  of  the  LVN  LLI  to  Intermediate-Level  Education  (ILE)  conducted  in 
residence  at  Fort  Leavenworth.  Surveys  therefore  were  administered  to  curriculum  developers, 
instructors,  and  students  directly  involved  with  resident  ILE  (2006  February-start  and  2006 
August- start  classes).  Some  interviews  also  were  conducted  with  ILE  instructors  at  satellite 
campuses  and  at  Total  Army  School  System  (TASS)  Battalion  sites. 
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Overall,  the  evaluation  results  indicated  that  the  LVN  LLI  is  a  cost-effective  solution  for 
enhancing  the  educational  outreach  of  ILE  curriculum  materials.  It  currently  costs  approximately 
$163.14  more  per  student  to  deliver  ILE  using  the  LVN  LLI  but  the  increase  in  cost  due  to 
technology  procurement  and  associated  personnel  and  facilities  expenses  is  a  vanishingly  small 
fraction  of  the  total  amount  spent  to  deliver  ILE.  Moreover,  the  relative  cost-per-student  to  the 
Army  will  further  decrease  as  the  printing  and  shipping  of  course  materials  to  remote  sites  is 
completely  phased  out  over  the  next  two  years.  For  what  amounts  practically  to  the  same  amount 
of  money,  ILE  delivered  online  using  the  LVN  LLI  is  in  the  process  of  eliminating  the  one-  to 
three- year  lag  in  curriculum  content  between  the  schoolhouse  and  Army  reserve  facilities.  It  also 
already  supports  anytime,  anywhere  learning  for  ILE  students  in  residence  and  at  satellite 
campuses,  as  reflected  in  system  usage  data. 

Unfortunately,  there  was  relatively  little  the  LVN  LLI  could  do  to  reduce  the  6-10  month 
lag  between  changes  in  the  operational  environment  and  revisions  to  the  standardized  ILE 
curriculum  due  to  the  lengthy  institutional  curriculum  review,  revision,  and  vetting  process. 
However,  the  LVN  LLI  may  assist  in  circumventing  this  problem  by  making  it  easier  for 
instructors  to  supplement  curriculum  materials  with  up-to-date  articles,  professional  discussions, 
and  emerging  doctrine.  Twenty-nine  percent  of  ILE  instructors  surveyed  reported  that  they 
supplemented  the  curriculum  materials  more  frequently  than  in  pre-LVN  LLI  conditions.  This 
proportion  is  likely  to  increase  as  faculty  become  more  facile  with  Blackboard.  In  addition, 
substantial  proportions  of  students  (>85%),  instructors  (74%),  and  course  authors  (80%)  reported 
that  the  ILE  course  content  was  relevant  to  the  jobs  of  ILE  graduates. 

There  did  appear  to  be  some  challenges  to  the  full-scale  adoption  of  the  LVN  LLI  by 
course  authors,  instructors,  and  students  in  the  schoolhouse.  Although  the  rates  of  active 
adoption  (i.e.,  primary  use  of  the  system  components)  increased  during  the  first-year  pilot,  they 
remain  below  100%.  A  combination  of  survey  and  interview  data  indicated  that  technical 
difficulty  was  not  the  main  barrier  to  system  use.  First,  a  weighted  average  of  student,  course 
author,  and  instructor  survey  responses  indicated  that  the  LVN  LLI  technical  support  was 
accessible  (62%)  and  useful  (66%).  Moreover,  the  system  workarounds  they  reported  using 
involved  typically  some  other  form  of  technology.  For  example,  SharePoint  was  the  most 
frequently  cited  alternative  to  Blackboard  when  posting  or  accessing  curriculum  materials. 

The  main  barriers  to  system  use  appear  to  have  been  (a)  ready  availability  of  alternatives 
to  the  system  that  were  more  familiar  and  easier  to  access;  (b)  inconsistent  or  absent  information 
management  procedures  necessary  to  make  posted  content  easy  to  find;  and  (c)  lack  of  instructor 
and  course  author  involvement  in  and  buy-in  to  the  lifelong  learning  concept.  As  has  been  found 
in  previous  evaluations  of  this  kind  (Cianciolo,  Heiden,  &  Prevou,  2006),  greater  success  has 
been  achieved  in  the  technical  implementation  of  the  initiative  than  in  the  cultivation  of 
stakeholder  enthusiasm  and  investment.  When  asked  what  they  thought  the  Army’s  main  reason 
for  implementing  Blackboard  and  Sharepoint  was,  a  minority  of  students  surveyed  (33%) 
responded  that  the  Army  had  made  the  decision  in  the  best  interests  of  leader  education. 

Although  the  Army  surely  had  multiple  reasons  for  adopting  the  new  instructional  technologies— 
including  enhancing  leader  education— actively  using  the  LVN  LLI  to  foster  students’  perception 
of  organizational  support  for  leaders  would  make  it  a  more  effective  asset  in  enhancing 
organizational  commitment  and  cultural  shift. 
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In  any  case,  challenges  to  the  active  adoption  of  the  LVN  LLI  did  not  hamper  the 
academic  experience  of  ILE  students  and  instructors  in  the  first- year  pilot.  A  very  high 
proportion  of  surveyed  students  (79%  on  average)  reported  that  ILE  instructors  generally 
demonstrated  the  classroom  facilitation  behaviors  recognized  as  critical  to  developing  adult 
learners.  In  addition,  survey  responses  indicated  that  a  majority  of  students  (a)  took 
responsibility  for  their  own  learning  via  independent  study  (70-98%);  (b)  reported  high  learning 
self-efficacy  (88%);  and  (c)  were  motivated  to  learn  the  topics  taught  in  ILE  (91%).  While  fifty- 
eight  percent  of  instructors  reported  that  the  LVN  LLI  did  not  have  an  impact  on  classroom 
efficiency  a  small  minority  of  instructors  (10%)  reported  that  the  LVN  LLI  enhanced  their 
classroom  efficiency.  Although  it  may  seem  insignificant,  this  result  is  noteworthy  given  the  fact 
that  the  introduction  of  instructional  technology  often  is  associated  with  increased  workload  for 
faculty  (Willis,  2000). 

In  order  for  the  already  successful  LVN  LLI  to  achieve  optimal  educational  and 
organizational  impact,  stronger  beliefs  in  its  purpose  and  higher  active  adoption  rates  of  its 
components  should  be  achieved.  The  accomplishment  of  this  goal  requires  fostering  the 
instructor  buy-in  necessary  to  learn  and  adopt  unfamiliar  technologies  and  to  role-model  their 
use  for  students.  Improved  buy-in  maybe  developed  through  (a)  engaging  other  schoolhouse 
components,  especially  faculty  and  staff  development,  in  viewing  and  cultivating  LVN  LLI- 
assisted  teaching  as  a  critical  instructor  competency;  (b)  encouraging  and  shepherding  the 
involvement  of  instructors  and  course  authors  in  the  development  of  information  management 
procedures  through  needs  assessment  and  iterative  system  design;  and  (c)  spreading  strategic 
communications  that  explain  the  purpose  and  goals  of  the  system,  that  anticipate  technical 
limitations  of  the  system  (i.e.,  login  requirements),  and  that  immediately  follow  system  outages. 

CONCLUSION 

The  present  research  represents  the  first  attempt  to  integrate  the  widely  diffuse  literature 
on  program  evaluation,  blended  learning,  organizational  effectiveness,  and  military  readiness  in 
order  to  conduct  an  evaluation  of  an  Army  technology-assisted  learning  initiative.  The  basis  of 
the  LLC  assessment  framework  in  theory  makes  it  generalizable  not  only  across  current  and 
future  LLCs,  but  also  across  blended  learning  initiatives  more  generally,  addressing  a  gap  in  the 
scholarly  literature  with  regard  to  the  effectiveness  assessment  of  educational  technology. 

Review  of  the  assessment  framework  by  key  stakeholders  from  each  of  the  pilot  LLCs  ensured 
generalizability  across  the  different  initiatives.  The  assessment  framework  developed  as  part  of 
this  investigation  proved  to  be  a  feasible  and  useful  tool  for  capturing  and  representing  the 
functioning  of  blended  learning  initiatives  such  as  LLCs,  from  the  acquisition  and  use  of 
resources  to  the  achievement  of  organizational  impact. 

This  research  is  not  without  its  limitations,  however.  First,  strategies  for  evaluating 
educational  effectiveness  have  historically  assumed  that  institutional  goals  for  education  are 
aligned  both  with  student  learning  objectives  and  with  those  learning  objectives  that  must  be  met 
to  further  society  or  organizational  performance.  The  design  of  the  LLC  assessment  framework 
is  consistent  with  this  assumption.  That  is,  outcome  measures  reflecting  the  alignment  (i.e., 
content  coverage)  between  what  is  taught  via  LLCs  and  what  must  be  taught  in  order  to  advance 
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student’s  expected  achievement  and  organizational  excellence  were  not  included  in  the 
framework.  This  alignment  is  critical  to  achieving  organizational  impact  via  an  educational 
initiative  (see  e.g.,  Cianciolo  et  al.,  2006),  and  should  be  included  in  a  comprehensive  picture  of 
LLC  effectiveness.  Second,  the  limited  scope  of  the  investigation  prevented  an  explicit  test  of  the 
validity  of  the  assessment  framework  developed.  Additional  research  should  determine  whether 
the  framework  (a)  applies  as  expected  to  other  LLCs,  especially  those  with  curricula  that  differ 
from  Intermediate-Level  Education;  (b)  produces  the  same  results  using  quasi-experimental 
evaluation,  where  possible;  and  (c)  accurately  predicts  the  future  status  of  an  LLC  based  on 
initial  evaluation  and  implementation  of  recommendations. 

The  findings  of  the  Fort  Leavenworth  evaluation  indicated  the  importance  of  taking  a 
causal  approach  where  quasi-experimental  design  was  not  possible.  Assessment  of  outcomes 
alone  would  have  indicated  that  the  initiative  had  achieved  its  goals  but  would  have  obscured  the 
fact  that  a  subset  of  these  goals— teaching  and  learning  effectiveness-was  achieved  largely 
independently  of  the  use  of  blended  learning  technologies.  A  causal  assessment  framework  such 
as  the  one  researched  and  developed  in  the  present  investigation  verified  whether  the  apparent 
success  of  an  LLC  was  truly  driven  by  the  activities  and  outputs  generated  by  the  initiative.  It 
also  provided  clear  recommendations  for  bridging  gaps  between  resources,  personnel  activities, 
system  outputs,  and  outcomes  in  order  to  enhance  impact.  Future  LLCs  will  achieve  the  greatest 
success  by  engaging  all  stakeholders  in  the  design  and  implementation  process  in  order  to  win 
the  buy-in  of  instructors,  other  staff,  and  learners.  The  enhanced  functioning  of  the  initiative 
resulting  from  stakeholder  engagement  will  enable  the  LLC  to  serve  as  a  powerful  driver  of 
lifelong  learning  and  organizational  change. 
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APPENDIX  A 

LLC  COMPREHENSIVE  METRIC  AND  MEASURE  CHART 
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Applicable  To 

Metric  Category  Metric  (Course  location;  Data  Collection  LVN  MANSCEN 

_ Course  topics) 

/o  student  users  reporting  education-  All  locations;  All  topics  Student  survey  (single 

related  problems  due  to  at-home  item) 
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Applicable  To 

Metric  Category  Metric  (Course  location;  Data  Collection  LVN  MANSCEN  UIT 

Course  topics) 

Perceived  leader  All  locations;  Topics  N/A  Course  production  team 

effectiveness  survev  ('mnltinlp 


for  leveraging  the 
capabilities  of  system 
components 


Applicable  To 

Metric  Category  Metric  (Course  location;  Data  Collection  LVN  MANSCEN  UIT 

_ Course  topics) _ 

Resources:  Money  —  Cost  per  Course  (Summed  categories  equal  total  cost  for  all  courses)  (For  cost  comparisons,  the  analogous  metrics  must  be 
used  for  the  comparison  condition.) 
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APPENDIX  B 

EVALUATION  OF  THE  FORT  LEAVENWORTH  LIFELONG  LEARNING  INITIATIVE 

Executive  Summary 

Overview 

The  Fort  Leavenworth  Lifelong  Learning  Initiative  (LVN  LLI)  is  a  suite  of  technologies 
(Blackboard,  Microsoft  Sharepoint,  and  Macromedia  Breeze)  that  enables,  among  other  things, 
online  posting  of  schoolhouse  curriculum  materials  and  collaboration  (asynchronous  and 
synchronous)  among  technical  staff,  students,  faculty,  and  curriculum  developers.  The  primary 
goal  of  this  blended  learning  initiative  is  to  provide  current,  standardized  curriculum  materials  24 
hours  a  day,  7  days  a  week,  to  students  located  in  the  schoolhouse,  on  satellite  campuses,  at 
Army  reserve  facilities,  and  at  personal  sites.  The  LVN  LLI  also  is  intended  to  support  two-way 
collaboration  between  the  field  and  institutional  Army,  enabling  enhanced  course  relevance  and 
graduate  reachback  to  the  institution. 

The  purpose  of  the  present  evaluation  was  to  capture  the  effectiveness  of  the  LVN  LLI 
when  applied  to  delivering  Intermediate-Level  Education  (ILE).  Specifically,  this  formative 
evaluation  attempted  to  define  and  measure  such  program  outcomes  as  enhanced 
teaching/learning  environment ,  enhanced  course  relevance ,  improved  student  performance,  and 
cost-effectiveness.  Factors  theoretically  linked  to  these  outcomes-chiefly  system  use  and  the 
activities  of  instructors,  curriculum  developers,  technical  support,  and  program  leadership— also 
were  assessed  in  order  to  tell  a  complete  story  about  the  functioning  of  the  initiative.  This 
appendix  presents  the  evaluation  in  detail,  with  the  main  findings  summarized  immediately 
below. 

Data  Collection 

Following  the  completion  of  a  program  logic  model  and  metric  development,  data  were 
collected  during  the  months  of  September  and  October  2006  using  a  variety  of  methods  that 
included  surveys  and  interviews,  system  analysis,  and  retrieval  of  archival  financial  and 
enrollment  data.  Data  collection  largely  focused  on  the  application  of  the  LVN  LLI  to  ILE 
conducted  in  residence  at  Fort  Leavenworth,  as  this  was  the  focus  prescribed  by  the  project 
stakeholders.  Surveys  therefore  were  administered  to  curriculum  developers,  instructors,  and 
students  directly  involved  with  resident  ILE  (2006  February-start  and  2006  August-start  classes; 
see  Appendices  B-l,  B-2,  and  B-3  for  the  surveys  and  a  description  of  their  administration). 
Some  interviews  also  were  conducted  with  ILE  instructors  at  satellite  campuses  and  at  Total 
Army  School  System  (TASS)  Battalion  sites. 

Findings 

Overall,  the  evaluation  results  indicate  that  the  LVN  LLI  is  a  cost-effective  solution  for 
enhancing  the  educational  outreach  of  ILE  curriculum  materials.  It  currently  costs  approximately 
$163.14  more  per  student  to  deliver  ILE  using  the  LVN  LLI  but  the  increase  in  cost  due  to 
technology  procurement  and  associated  personnel  and  facilities  expenses  is  a  vanishingly  small 
fraction  of  the  total  amount  spent  to  deliver  ILE.  Moreover,  the  relative  cost-per-student  to  the 
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Army  will  further  decrease  as  the  printing  and  shipping  of  course  materials  to  remote  sites  is 
completely  phased  out  over  the  next  two  years.  For  what  amounts  practically  to  the  same  amount 
of  money,  ILE  delivered  online  using  the  LVN  LLI  is  in  the  process  of  eliminating  the  one-  to 
three- year  lag  in  curriculum  content  between  the  schoolhouse  and  Army  reserve  facilities.  It  also 
supports  anytime,  anywhere  learning  for  ILE  students  in  residence  and  at  satellite  campuses,  as 
reflected  in  system  usage  data. 

Unfortunately,  there  is  relatively  little  the  LVN  LLI  can  do  to  reduce  the  6-10  month  lag 
between  changes  in  the  operational  environment  and  revisions  to  the  standardized  ILE 
curriculum  due  to  the  lengthy  institutional  curriculum  review,  revision,  and  vetting  process. 
However,  the  LVN  LLI  may  assist  in  circumventing  this  problem  by  making  it  easier  for 
instructors  to  supplement  curriculum  materials  with  up-to-date  articles,  professional  discussions, 
and  emerging  doctrine.  Twenty-nine  percent  of  ILE  instructors  surveyed  reported  that  they 
supplemented  the  curriculum  materials  more  frequently  than  in  pre-LVN  LLI  conditions.  This 
proportion  is  likely  to  increase  as  faculty  become  more  facile  with  Blackboard.  In  addition, 
substantial  proportions  of  students  (>85%),  instructors  (74%),  and  course  authors  (80%) 
surveyed  reported  that  the  ILE  course  content  was  relevant  to  the  jobs  of  ILE  graduates. 

There  do  appear  to  be  some  challenges  to  the  full-scale  adoption  of  the  LVN  LLI  by 
course  authors,  instructors,  and  students  in  the  schoolhouse.  Although  the  rates  of  active 
adoption  (i.e.,  primary  use  of  the  system  components)  increased  during  the  first-year  pilot,  they 
remain  below  100%  (between  5%  and  40%,  depending  on  the  system  component  and  purpose  for 
which  it  was  accessed).  A  combination  of  survey  and  interview  data  indicated  that  technical 
difficulty  was  not  the  main  barrier  to  system  use.  First,  a  weighted  average  of  student,  course 
author,  and  instructor  survey  responses  indicated  that  the  LVN  LLI  technical  support  was 
accessible  (62%)  and  useful  (66%).  Moreover,  the  system  workarounds  they  reported  using 
involved  typically  some  other  form  of  technology.  For  example,  SharePoint  was  the  most 
frequently  cited  alternative  to  Blackboard  when  posting  or  accessing  curriculum  materials. 

The  main  barriers  to  system  use  appear  to  have  been  (a)  ready  availability  of  alternatives 
to  the  system  that  were  more  familiar  and  easier  to  access;  (b)  inconsistent  or  absent  information 
management  procedures  necessary  to  make  posted  content  easy  to  find;  and  (c)  lack  of  instructor 
and  course  author  involvement  in  and  buy-in  to  the  lifelong  learning  concept.  As  has  been  found 
in  previous  evaluations  of  this  kind  (Cianciolo  et  al.,  2006),  greater  success  has  been  achieved  in 
the  technical  implementation  of  the  initiative  than  in  the  cultivation  of  stakeholder  enthusiasm 
and  investment.  When  asked  what  they  thought  the  Army’s  main  reason  for  implementing 
Blackboard  and  Sharepoint  was,  a  minority  of  students  surveyed  (33%)  responded  that  the  Army 
had  made  the  decision  in  the  best  interests  of  leader  education.  Although  the  Army  surely  had 
multiple  reasons  for  adopting  the  new  instructional  technologies— including  enhancing  leader 
education— actively  using  the  LVN  LLI  to  foster  students’  perception  of  organizational  support 
for  leaders  would  make  it  a  more  effective  asset  in  enhancing  organizational  commitment  and 
cultural  shift. 

In  any  case,  challenges  to  the  active  adoption  of  the  LVN  LLI  did  not  hamper  the 
academic  experience  of  ILE  students  and  instructors  in  the  first-year  pilot.  A  very  high 
proportion  of  surveyed  students  (79%  on  average)  reported  that  ILE  instructors  generally 
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demonstrated  the  classroom  facilitation  behaviors  recognized  as  critical  to  developing  adult 
learners.  In  addition,  survey  responses  indicated  that  a  majority  of  students  (a)  took 
responsibility  for  their  own  learning  via  independent  study  (70-98%);  (b)  reported  high  learning 
self-efficacy  (88%);  and  (c)  were  motivated  to  learn  the  topics  taught  in  ILE  (91%).  Fifty-eight 
percent  of  instructors  reported  that  the  LVN  LLI  did  not  have  an  impact  on  classroom  efficiency, 
and  a  small  minority  of  instructors  (10%)  reported  that  the  LVN  LLI  enhanced  their  classroom 
efficiency.  Although  it  may  seem  insignificant,  this  result  is  noteworthy  given  the  fact  that  the 
introduction  of  instructional  technology  often  is  associated  with  increased  workload  for  faculty 
(Willis,  2000). 

Conclusions  and  Recommendations 

If  nothing  about  the  LVN  LLI  implementation  were  to  change  from  this  point  forward, 
ILE  instruction  would  continue  successfully,  and  students— especially  those  at  Army  reserve 
facilities— would  benefit  from  greater  access  to  more  relevant  standardized  curriculum,  all  at 
relatively  little  cost  to  the  Army.  However,  the  findings  of  the  present  evaluation  illustrate  the 
importance  of  taking  a  logic  modeling  approach  to  understanding  the  impact  of  lifelong  learning 
initiatives.  Assessment  of  outcomes  alone  would  have  indicated  that  the  LVN  LLI  had  achieved 
its  goals  but  would  have  obscured  the  fact  that  a  subset  of  these  goals— teaching  and  learning 
effectiveness— was  achieved  independently  of  the  use  of  the  LVN  LLI’s  components. 

In  order  for  the  already  successful  LVN  LLI  to  achieve  optimal  educational  and 
organizational  impact,  stronger  beliefs  in  its  purpose  and  higher  active  adoption  rates  of  its 
components  should  be  achieved.  The  accomplishment  of  this  goal  requires  fostering  the 
instructor  buy-in  necessary  to  learn  and  adopt  unfamiliar  technologies  and  to  role-model  their 
use  for  students.  Improved  buy-in  may  be  developed  through  (a)  engaging  other  schoolhouse 
components,  especially  faculty  and  staff  development,  in  viewing  and  cultivating  LVN  LLI- 
assisted  teaching  as  a  critical  instructor  competency;  (b)  encouraging  and  shepherding  the 
involvement  of  instructors  and  course  authors  in  the  development  of  infonnation  management 
procedures  through  needs  assessment  and  iterative  system  design;  and  (c)  spreading  strategic 
communications  that  explain  the  purpose  and  goals  of  the  system,  that  anticipate  technical 
limitations  of  the  system  (i.e.,  login  requirements),  and  that  immediately  follow  system  outages. 
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Overview  of  the  Fort  Leavenworth  Lifelong  Learning  Initiative 

The  Fort  Leavenworth  Lifelong  Learning  Initiative  (LVN  LLI)  was  designed  to  support 
the  educational  mission  of  the  U.S.  Army  Command  and  General  Staff  College  (CGSC).  The 
CGSC  is  one  of  the  major  subordinate  organizations  comprising  the  Combined  Arms  Center 
headquartered  at  Fort  Leavenworth,  and  provides  formal  institutional  education  to  field  grade 
and  higher  officers  through  five  component  schools.  Figure  B-l  shows  the  organizational  chart 
of  the  CGSC,  adapted  from  CGSC  Circular  350-1.  The  components  of  the  CGSC  directly 
relevant  to  the  present  evaluation  are  shown  in  orange,  and  their  connection  with  the  LVN  LLI 
will  be  explained  in  further  detail  below. 


Figure  B-l.  CGSC  Organizational  Chart 
The  Components  of  the  LVN  LLI 

Technology  procurement  and  integration  for  the  LVN  LLI  began  in  2004,  and  at  the  time 
of  this  evaluation  the  initiative  comprised  three  major  components:  Blackboard,  Microsoft 
SharePoint,  and  Macromedia  Breeze.  Each  of  these  components,  described  below,  was  selected 
to  provide  a  unique  capability  in  supporting  the  development  of  curriculum  materials,  the 
delivery  of  course  content,  the  collaboration  among  faculty,  staff,  and  students,  and  the  storage 
of  curriculum  materials  for  reachback  from  graduates  in  the  field. 
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Blackboard.  The  Blackboard  Academic  Suite  is  a  web-based  platform  designed  to 
organize  and  host  a  wide  variety  of  file  types  typically  distributed  in  educational  settings, 
including  (but  not  limited  to)  digital  documents,  presentations,  and  videos.  Blackboard  also 
provides  a  means  for  faculty  to  communicate  with  students  via  posting  announcements  and 
sending  group  emails  and  to  conduct  such  classroom  administrative  tasks  as  administering 
exams,  collecting  assignments,  and  recording  grades.  Students  may  also  communicate  with  each 
other  using  Blackboard’s  discussion  board.  Blackboard  is  especially  well  suited  for  conducting 
asynchronous  collaborative  distance-learning  courses,  such  as  those  provided  by  many 
universities,  and  for  conducting  blended-leaming  courses  in  which  one  component  involves 
asynchronous  distance  learning.  Online  posting  of  course  content  also  reduces  the  time  and 
money  involved  in  replicating  and  shipping  curriculum  materials  to  multiple  class  sites,  a  benefit 
of  particular  relevance  to  the  CGSC. 

Microsoft  SharePoint.  A  combination  of  Microsoft  Windows  server  technology  and 
access  privileges,  SharePoint  enables  the  intranet-based  storage  and  management  of  large, 
complex  file  collections.  SharePoint  users  with  the  appropriate  permissions  and  access  to  an 
organization’s  domain  can  create  websites  ranging  in  complexity  to  host  and  organize  content, 
post  announcements,  create  and  maintain  message  boards,  and  send  emails,  among  other  things. 
SharePoint  is  intended  to  support  asynchronous  collaboration  among  curriculum  developers  by 
making  working  draft  files  and  reference  information  available  in  a  way  that  is  easy  to  locate  and 
understand.  SharePoint  also  is  intended  to  support  collaboration  among  students,  to  distribute 
general  reference  materials  (e.g.,  emerging  doctrine)  and  to  connect  students  administratively  to 
the  schoolhouse. 

Macromedia  Breeze.  Breeze  is  a  web-conferencing  platform  that  enables  synchronous 
collaboration  through  such  capabilities  as  document  sharing,  voiceover  IP,  multiple-person 
video,  and  an  electronic  whiteboard.  Breeze  was  included  in  the  LVN  LLI  in  large  part  to  enable 
advanced  user  technical  support  and  to  enable  distributed  academic  activities,  such  as  instructor 
or  course  author  meetings  and  virtual  classroom  presentations  and  group  exercises.  At  the  time 
of  this  research,  Breeze  was  in  the  preliminary  stages  of  implementation  in  the  LVN  LLI,  so  it  is 
not  addressed  further  in  this  report. 

Implementation,  maintenance,  and  oversight  of  the  LVN  LLI  are  conducted  by  the  CGSC 
Directorate  of  Educational  Technology  (DOET;  highlighted  in  orange  in  Figure  B-l  above).  The 
DOET  comprises  1 8  staff  members  whose  efforts  are  devoted  to  providing  the  information 
technology  necessary  to  support  the  CGSC’s  educational  mission,  which  includes  classroom  and 
office  networking  and  automation  in  addition  to  the  LVN  LLI.  With  the  Lewis  and  Clark 
Building— which  will  house  CGSS  and  much  of  the  CGSC  administration  and  support,  including 
the  DOET— under  construction,  the  DOET  also  is  heavily  involved  in  planning,  procuring,  and 
overseeing  the  incorporation  of  advanced  information  technology  into  over  365,000  square  feet 
of  brand  new  classrooms,  offices,  and  auditorium  space.  Eight  additional  staff  members,  on-site 
contractors,  are  devoted  entirely  to  the  LVN  LLI,  providing  technical  support,  technology 
integration,  and  user  training. 
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The  L  VN  LLI  and  Officer  Education 


Shown  at  the  bottom  of  Figure  B-l  (above)  are  the  five  schools  comprising  the  CGSC. 
Highlighted  in  orange  to  indicate  their  direct  relevance  to  the  present  evaluation  are  the 
Command  and  General  Staff  School  (CGSS)  and  the  School  of  Advanced  Distance  Learning 
(SADL).  Together,  these  two  schools  deliver  the  common  core  or  Intermediate-Level  Education 
(ILE)  component  of  the  Command  and  General  Staff  Officer’s  Course  (CGSOC).  The  CGSS 
delivers  ILE  to  students  in  residence  at  Fort  Leavenworth  and  to  students  attending  satellite 
campuses  located  at  Forts  Belvoir,  Gordon,  and  Lee.  The  SADL  develops,  distributes,  and 
administers  classroom-based  ILE  to  students  attending  instruction  via  Total  Army  School 
System  (TASS)  Battalion  sites  and  self-contained,  web/CD-based  ILE  instruction  to  non-resident 
students  completing  their  coursework  in  other  locations.  The  present  evaluation  focused  on  the 
administration  of  ILE  using  the  LVN  LLI  because  at  the  time  the  study  was  conducted,  ILE  was 
the  major  course  supported  by  the  initiative.  The  CGSC  is  in  the  process  of  implementing  other 
courses  online,  such  as  those  administered  by  the  School  of  Advanced  Military  Studies. 

The  ILE  curriculum  predates  the  LVN  LLI  by  approximately  two  years.  First  piloted  in 
2003,  the  purpose  of  ILE  was  to  make  third  tier  officer  education  available  to  all  Army  majors  in 
a  manner  best  suited  to  their  career  field  (Bralley,  2006;  Bralley,  Danley,  French,  Soby,  & 

Tiberi,  2003).  Prior  to  ILE,  50%  of  Army  majors  were  selected  to  attend  the  10-month  CGSOC 
at  Fort  Leavenworth.  The  selected  majors  came  from  a  variety  of  career  fields,  however  a 
substantial  portion  of  the  CGSOC  curriculum  was  relevant  only  to  a  subset  of  these  officers. 
Majors  in  the  Operations  Career  Field  benefited  from  the  complete  curriculum,  but  majors  in 
other  career  fields  (e.g.,  the  Institutional  Support  Career  Field)  found  only  the  common  core 
component  to  be  relevant  to  their  educational  needs.  Enabled  by  the  satellite  campuses  and 
advances  in  distance  learning  technology,  ILE  is  an  efficient  and  cost-effective  solution  for 
ensuring  educational  outreach.  Majors  in  the  Operations  Career  Field  attend  the  full  10-month 
course  at  Fort  Leavenworth,  while  majors  in  the  other  career  fields  complete  only  the  CGSOC 
common  core  curriculum  (i.e.,  ILE),  lasting  approximately  sixteen  weeks,  at  one  of  the  satellite 
campuses  or  via  distance  education  (Bralley,  2006;  Bralley  et  al.,  2003). 

As  shown  in  Figure  B-2  below  (adapted  from  CGSC  Circular  350-1),  the  ILE  curriculum 
is  taught  in  two  parallel  sequences  of  blocks  of  instruction  punctuated  by  periodic  visits  by  guest 
speakers  to  CGSC.  All  ILE  students,  regardless  of  their  course  location,  receive  this  same  basic 
curriculum,  although  the  structure  and  duration  of  the  course  differs  somewhat  for  distance 
learners,  and  guest  lectures  must  be  attended  remotely  or  asynchronously  by  students  located  off- 
post. 


Standardized  curriculum  materials  for  ILE  are  developed  by  teams  of  lesson  authors 
grouped  according  to  block  of  instruction.  Block  authors  lead  a  team  of  lesson  authors,  ensuring 
that  the  content  developed  is  accurate  and  coherent.  Curriculum  revisions  made  following  an 
administration  of  ILE  go  through  a  rigorous  development  process  that  begins  with 
recommendations  gathered  from  a  post-instructional  conference.  A  curriculum  design  review 
involving  the  college  leadership  precedes  the  revision  process  then,  prior  to  distribution  to 
students,  revised  ILE  curriculum  materials  undergo  an  institutional  review  process  whereby 
accuracy  is  further  checked  and  instructional  effectiveness  examined. 
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-C  Guest  Speakers 
Figure  B-2.  ILE  Curriculum 


ILE  students  in  residence  at  Fort  Leavenworth  and  at  satellite  campuses  are  divided  into 
sections  of  approximately  72  students,  which  are  in  turn  divided  into  four  staff  groups 
comprising  approximately  18  students.  Each  section  is  taught  by  12  instructors  coming  from 
each  of  five  departments  within  CGSS:  Tactics  (4  instructors),  Joint  and  Multinational 
Operations  (4  instructors),  Logistics  and  Resource  Operations  (2  instructors),  Military  History  (1 
instructor),  and  Leadership  (1  instructor).  Staff  groups  contained  in  a  section  have  one  instructor 
from  each  department  such  that  a  section’s  four  tactics  instructors  each  have  responsibility  for  a 
single  staff  group  whereas  a  section’s  single  military  history  instructor  has  responsibility  for  all 
four  staff  groups,  and  so  on. 


Producing  and  Delivering  ILE  with  the  LVNLLI-  Use  Cases 

ILE  Curriculum  Development.  SharePoint  is  used  in  the  LVN  LLI  to  store  and  manage 
ILE  curriculum  content  during  the  development  process.  Block  authors  are  encouraged  to  use 
SharePoint  to  coordinate  with  each  other  and  with  lesson  authors  so  that  content  is  not 
unnecessarily  duplicated  and  so  that  general  educational  themes  (e.g.,  critical  thinking)  common 
across  blocks  may  be  reinforced  in  topic-relevant  ways.  Block  authors  have  permissions  to  create 
and  modify  SharePoint  websites  to  coordinate  their  team’s  efforts. 

Prior  to  the  implementation  of  the  LVN  LLI,  ILE  curriculum  materials  under 
development  were  stored  on  an  internal  network  and  block  authors  created  file  systems  to 
organize  content.  Replacing  the  internal  network  with  SharePoint  enables  block  authors  and 
lesson  authors  to  work  together  using  a  central  information  management  resource  even  when 
they  are  geographically  distributed  on-  and  off-post.  The  SharePoint  site  also  links  curriculum 
developers  to  a  variety  of  knowledge  management  resources,  including  the  Battle  Command 
Knowledge  System  discussion  forums  and  Army  Knowledge  Online  (AKO),  so  that  information 
directly  from  the  field  is  readily  available  and  can  be  used  to  enhance  the  currency  and  relevance 
of  course  content. 


ILE  Instruction.  The  intended  use  for  Blackboard  is  to  deliver  standardized  ILE 
curriculum  materials  to  students  located  on-  and  off-post.  Secured  by  AKO  authentication, 
content  posted  on  Blackboard  is  accessible  anytime,  anywhere  by  students  and  faculty  with  the 
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appropriate  permissions.  Because  the  curriculum  development  and  review  process  creates  a  gap 
of  several  (up  to  1 0)  months  between  the  initiation  of  the  process  and  content  delivery,  ILE 
instructors  are  encouraged  to  supplement  the  formal  curriculum  with  material  they  identify  as 
more  relevant  or  current.  Instructors  supplementing  the  formal  curriculum  may  locate  materials 
using  readily  available  links  to  Army  knowledge  management  resources.  They  are  directed  to 
post  their  content  on  Blackboard  and  to  alert  students  to  new  materials.  Instructors  also  are 
encouraged  to  use  the  administrative  functions  present  in  Blackboard,  particularly  the  grade 
book,  to  facilitate  classroom  activity. 

Prior  to  the  implementation  of  the  LVN  LLI,  instructors  distributed  curriculum  materials 
via  an  internal  network  and  via  print.  Access  to  course  content  therefore  was  restricted  to  people 
on  post  who  could  access  computers  on  the  network,  unless  substantial  printing  expenses  were 
incurred  to  facilitate  home  study.  Remote  ILE  students  also  received  curriculum  materials  in 
printed  fonn  and,  more  recently,  via  compact  disc.  Posting  course  content  in  Blackboard  instead 
of  on  an  internal  network  enables  access  to  course  content  from  anywhere.  Web-based  access  to 
course  content  not  only  enables  the  delivery  of  standardized  course  content  across  modes  of 
delivery  (resident,  satellite,  TASS  Battalion,  etc.),  but  eliminates  the  need  for  printing  and 
shipping  to  remote  sites.  Eliminating  printing  and  shipping  simultaneously  reduces  cost  and 
enhances  curriculum  currency  for  students  off-post  because  the  lead  time  to  reproduce  and 
distribute  course  materials  is  significantly  reduced. 

Student  collaboration.  In  addition  to  supporting  collaboration  among  curriculum 
developers,  SharePoint  also  is  used  to  support  ILE  student  collaboration  and  connection  to  the 
schoolhouse.  The  student  SharePoint  site  features  such  resources  as  reference  libraries  and  links, 
academic  calendars,  and  announcements  from  a  variety  of  sources,  including  the  deputy 
commandant  and  the  CGSC  student  division.  Students  with  the  appropriate  permissions  may 
create  shared  workspaces  on  the  site  to  post  their  contributions  to  class  discussions,  group 
projects,  and/or  practical  exercises.  Enabling  collaboration  within  a  protected  domain  facilitates 
information  assurance  in  an  environment  where  a  large  number  of  people  may  post  content. 
Unlike  in  SharePoint,  students  do  not  have  permissions  to  post  content  to  Blackboard  so  that 
access  controls  may  promote  information  assurance. 

Metrics  and  Measures  Used  to  Evaluate  the  Leavenworth  LLI 

The  purpose  of  the  evaluation  was  to  capture  the  learning  effectiveness  of  the  LVN  LLI. 
The  goal  of  this  evaluation  was  to  represent  the  cost-benefit  tradeoffs  of  the  emerging  initiative 
in  a  way  that  would  be  easy  for  decision  makers  to  understand  and  use.  Specifically,  this 
fonnative  evaluation  attempted  to  define  and  measure  such  outcomes  as  enhanced 
teaching/leaming  environment,  enhanced  course  relevance,  improved  student  performance,  and 
cost-effectiveness.  Factors  theoretically  linked  to  these  outcomes,  chiefly  system  use  and  the 
activities  of  instructors,  curriculum  developers,  technical  support,  and  leadership,  also  were 
assessed  in  order  to  tell  a  complete  story  about  the  functioning  of  the  initiative. 

Of  the  229  metrics  listed  in  the  metric  and  measures  chart  (see  Appendix  A,  above),  1 7 
were  selected  by  the  CGSC  DOET  as  being  particularly  relevant  to  capturing  the  costs  and 
benefits  of  the  LVN  LLI.  Eight  of  these  metrics  were  activity  metrics,  three  were  output  metrics, 
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and  six  were  outcome  metrics.  No  resource  metrics  were  selected.  Additional  resource,  activity, 
and  outcome  metrics  from  the  metrics  and  measures  chart  were  used  to  supplement  the  17 
selected  because  they  (a)  would  provide  useful  diagnostic  and  outcome  information;  (b)  were 
necessary  for  capturing  cost-effectiveness;  and  (c)  were  feasible  to  capture  in  the  time  available. 

Evaluation  data  were  collected  in  September  and  October  of  2006  via  a  combination  of 
interviews,  surveys,  and  retrieval  of  archival  financial  and  other  data.  Data  collection  largely 
focused  on  the  application  of  the  LVN  LLI  to  resident  ILE,  as  this  was  the  focus  prescribed  by 
the  project  stakeholders.  Surveys  therefore  were  administered  to  curriculum  developers, 
instructors,  and  students  directly  involved  with  resident  ILE  (2006  February-start  and  2006 
August-start  classes2;  see  Appendices  B-l,  B-2,  and  B-3  for  a  complete  description  of  the 
surveys  and  their  administration).  All  metrics  used  in  the  present  evaluation  and  their  associated 
measures  are  described  below. 

Outcome  -  Course  Relevance 

Six  metrics  capturing  the  relevance  of  the  ILE  curriculum  materials  (formal  and  informal) 
were  measured.  These  metrics  were  selected  in  order  to  explore  a  key  expected  outcome  of  the 
LVN  LLI  —  that  course  content  would  be  more  relevant  because  (a)  curriculum  developers 
would  have  easier  access  to  the  field  Army  and  to  each  other,  enabling  more  frequent  updates  to 
standardized  curriculum  materials;  (b)  instructors  would  have  more  flexibility  to  insert 
supplemental  materials  into  the  curriculum  in  real  time,  enabling  more  frequent  updates;  and  (c) 
placing  content  online  would  eliminate  replication  and  shipping  timelines.  A  detailed  description 
of  each  metric  is  provided  below. 

1.  Frequency  of  curriculum  developer  updates  to  main  curriculum  relative  to  the  pre- 
LVN  LLI  situation.  As  described  previously,  formal  decisions  to  update  the  main,  standardized 
ILE  curriculum  follow  a  post-instruction  conference  and  curriculum  design  review.  Course 
authors  then  implement  these  decisions,  ideally  completing  the  production  and  editorial  process 
before  the  next  cycle  of  the  course.  Course  authors  were  asked  to  report  via  survey  how 
frequently  they  updated  the  main  ILE  curriculum  (relative  to  course  cycles)  based  on  current 
events  or  feedback  in  past  years,  then  were  asked  to  report  the  impact  that  SharePoint  has  had  on 
the  frequency  of  such  updates.  Eight  of  9  course  authors  who  reported  on  curriculum  updating 
indicated  that  in  the  past  they  infrequently  or  never  updated  the  curriculum.  Seven  of  the  9 
course  authors  also  indicated  that  SharePoint  had  not  had  an  impact  on  the  frequency  of 
updating— that  the  duration  of  the  curriculum  development  and  review  process,  rather  than 
technical  factors  determined  the  frequency  of  revisions. 

2.  Curriculum  developer  perceptions  of  main  curriculum  relevance.  Course  authors  were 
surveyed  to  capture  their  perceptions  of  the  relevance  of  the  standardized  curriculum  materials. 

A  single  question  asked  how  current  or  relevant  they  felt  the  curriculum  was  to  the  jobs  of  ILE 


2  The  first  full-scale  pilot  of  the  LVN  LLI  with  resident  students  was  conducted  with  the  2006  February-start  ILE 
class  with  follow-on  implementation  executed  for  the  2006  August-start  class.  Some  initial  piloting  was  done  with 
the  2005  August-start  ILE  class,  but  this  pilot  effort  was  a  scaled-down  version  of  the  2006  February-start  pilot  and 
also  was  completed  prior  to  the  beginning  of  the  present  evaluation. 
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graduates.  Of  the  10  curriculum  developers  who  responded  to  this  question,  8  indicated  that  the 
curriculum  content  was  either  totally  relevant  or  more  relevant  than  in  previous  years. 

3.  Frequency  of  instructor  updates  to  ongoing  course  materials.  As  described  previously, 
the  standardized  ILE  curriculum  posted  on  Blackboard  in  the  LVN  LLI  can  be  up  to  10  months 
behind  changes  in  the  contemporary  operating  environment.  To  enhance  the  currency  of  course 
materials,  instructors  may  use  Blackboard  to  distribute  news  articles,  professional  literature,  or 
emerging  doctrine,  among  other  things,  just  in  time  to  replace  older  readings  or  support  class 
discussion.  The  frequency  of  such  updates  was  assessed  by  asking  instructors  how  often  they 
made  them:  daily,  weekly,  every  couple  of  weeks,  monthly,  and  never.  Of  69  instructors  who 
responded  to  this  survey  question,  64%  reported  updating  daily  or  weekly. 

4.  Frequency  of  instructor  updates  to  ongoing  course  materials  relative  to  the  pre-LVN 
LLI  situation.  This  metric  was  captured  by  asking  instructors  to  indicate  how  frequently  they 
supplemented  the  standardized  curriculum  during  the  2006  February-  and/or  2006  August-start 
classes,  taught  using  the  LVN  LLI,  relative  to  previous  ILE  classes  for  which  an  internal  network 
was  used  to  distribute  course  content.  Of  the  63  instructors  who  responded  to  this  survey 
question,  29%  (N  =  18)  reported  making  updates  somewhat  or  a  lot  more  frequently  than  in 
previous  situations.  Fifty-six  percent  (35  of  63)  reported  that  their  frequency  of  updates  was 
approximately  the  same. 

5.  Instructor  perceptions  of  course  relevance  (main  and  supplemental  content). 

Instructors  were  surveyed  for  their  perceptions  of  the  relevance  of  the  standardized  and 
supplemental  curriculum  materials.  A  single  question  asked  how  current  or  relevant  instructors 
felt  the  main  and  supplemental  curriculum  was  to  the  jobs  of  ILE  graduates.  Of  the  65  instructors 
who  responded  to  this  question  42%  (N=21)  stated  that  the  curriculum  was  totally  relevant  or 
definitely  more  relevant  than  in  the  past.  Thirty-two  percent  (21  of  65)  stated  that  the  curriculum 
was  relevant  but  that  emphasis  is  rightly  placed  on  education,  rather  than  mission  preparation. 

6.  Student  perceptions  of  course  relevance  (main  and  supplemental  content).  Students 
were  surveyed  to  capture  their  perceptions  of  the  relevance  of  the  main  and  supplemental 
curriculum  materials.  Two  questions  asked  how  current  or  relevant  students  felt  the  main  and 
supplemental  curriculum  was  to  the  jobs  they  expected  to  assume  after  graduation.  Of  the  251 
students  who  responded  to  these  questions,  73%  (N  =  184)  felt  the  main  curriculum  was  either 
more  relevant  than  irrelevant  or  totally  relevant  to  the  jobs  they  expect  to  have  after  graduating. 
An  additional  16%  of  respondents  (N  =  40)  indicated  that  the  formal  curriculum  was  relevant, 
given  the  fact  that  institutional  education  was  supposed  to  be  “just-in-case,”  not  “just-in-time.” 
Sixty-six  percent  of  surveyed  students  (N=  165)  felt  that  the  supplemental  content  was  either 
more  relevant  than  irrelevant  or  totally  relevant.  An  additional  22%  of  respondents  (N=  54) 
indicated  that  the  supplemental  content  was  relevant,  given  its  academic  purpose. 

Summary.  Ideally,  the  opportunity  to  update  standardized  ILE  curriculum  materials 
would  occur  before  each  course  administration.  Updating  the  curriculum  this  frequently  would 
enable  the  institution  to  deliver  education  as  adaptively  as  possible  to  changes  in  the  operational 
environment.  There  is  significant  difficulty  associated  with  updating  the  curriculum  this  often, 
namely  the  length  of  the  curriculum  review,  revision,  vetting,  and  printing  process  relative  to  the 
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number  of  overlapping  course  administrations.  Simply  put,  the  1LE  course  administration  cycle 
outpaces  the  curriculum  development  cycle.  The  result  is  that  standardized  ILE  curriculum 
materials  delivered  to  August-start  resident  students  are  6  to  10  months  behind  the  changes  to  the 
current  operating  environment.  Non-resident  course  administrations  (i.e.,  at  TASS  Battalions)  are 
even  further  behind— up  to  three  additional  years— in  part  due  to  the  time  required  to  print  or 
replicate  and  ship  course  materials  throughout  the  U.S.  and  internationally. 

Unfortunately,  the  LVN  LLI  cannot  change  the  speed  of  the  formal  curriculum  review, 
revision,  vetting,  and  printing  process.  That  said,  substantial  proportions  of  students  (>85%), 
course  authors  (80%),  and  instructors  (74%)  surveyed  reported  that  the  ILE  course  content  was 
relevant  to  the  jobs  of  ILE  graduates.  It  is  unknown  how  pre-LVN  LLI  ratings  of  ILE  course 
relevance  would  compare  to  those  obtained  in  the  present  evaluation.  However,  29%  of 
instructors  surveyed  reported  that  they  supplemented  the  curriculum  materials  more  frequently 
than  in  pre-LVN  LLI  conditions,  and  56%  reported  making  approximately  the  same  number  of 
updates  in  both  conditions.  Importantly,  when  fully  implemented  and  adopted,  the  LVN  LLI  will 
eliminate  the  one-year  discrepancy  in  ILE  content  currency  between  TASS  Battalion  sites  and 
the  CGSS  due  to  the  replication  and  shipping  timelines  required  to  produce  and  distribute  course 
content  remotely. 

Outcome  -  Instructors  as  Facilitators  of  Adult  Learning 

Three  metrics  capturing  the  degree  to  which  ILE  instructors  leveraged  the  LVN  LLI 
blended  learning  environment  (i.e.,  the  combination  of  face-to-face  and  asynchronous 
collaborative  learning  settings)  to  facilitate  adult  learning  were  measured.  Proponents  of 
technology-assisted  learning  claim  that  technology  can  revolutionize  the  way  education  is 
conducted,  but  instructors  must  leverage  the  available  capabilities  to  achieve  an  impact  on 
student  learning  and  lifelong  learning  orientation  (e.g.,  Clark,  1994;  Ehrmann,  1994).  The 
purpose  of  the  selected  metrics  was  to  explore  whether  the  presence  of  the  LVN  LLI 
(specifically,  Blackboard)  has  had  an  impact  on  how  instruction  and  learning  is  accomplished  in 
ILE.  A  detailed  description  of  each  metric  is  provided  below. 

1.  Percent  of  students  who  reported  that  they  were  encouraged  to  contribute  materials  to 
the  course  curriculum.  Students  were  asked  to  report  via  survey  whether  any  of  their  instructors 
encouraged  them  to  contribute  to  the  course  content  in  their  respective  areas  of  expertise.  The 
range  in  possible  answers  included  “no  encouragement,”  “no  discouragement,”  and  “active 
encouragement.”  Of  the  251  students  who  responded  to  the  survey  question,  74%  (TV  =  186) 
indicated  that  instructors  actively  encouraged  contributions.  Sixty-eight  percent  of  February-start 
students  (54  of  80)  reported  active  encouragement,  whereas  78%  of  August-start  students  (131  of 
1 69)  made  the  same  report. 

2.  Percent  of  students  who  reported  that  instructors  encouraged  them  (or  demonstrated 
how)  to  use  resources  other  than  him/herself  for  class  work  or  other  purposes.  Students  surveyed 
were  asked  to  report  whether  any  of  their  instructors  encouraged  them  to  consider  learning 
resources  in  addition  to  him-  or  herself.  Respondents  could  select  “no  encouragement,”  “no 
discouragement,”  and  “active  encouragement.”  Of  the  250  students  who  responded  to  the  survey 
question,  65%  (TV=  163)  indicated  that  instructors  actively  encouraged  the  exploration  of 
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alternative  learning  resources.  Roughly  equal  proportions  of  February- start  and  August-start 
students  reported  active  encouragement  [67%  (53  of  79)  and  65%  (109  of  169),  respectively]. 

3.  Quality  of  real-time  instructor  facilitation  behaviors  in  the  classroom.  This  metric  was 
captured  through  a  five-part  survey  question  in  which  students  were  asked  to  report  the  relative 
number  of  instructors  who  engaged  in  five  classroom  facilitation  behaviors.  These  behaviors 
included:  (1)  provided  clear  guidance  regarding  expectations  and  performance  evaluation 
criteria;  (2)  provided  constructive  and  informative  performance  feedback;  (3)  presented  learning 
challenges  that  were  difficult  but  not  overwhelming;  (4)  assigned  group  tasks  that  had  well- 
defined  tasks  and  roles  for  each  group  member;  and  (5)  engaged  students  in  asking  questions  and 
reflecting  on  class  materials.  Table  B-l  below  summarizes  student  responses  to  this  question. 


Table  B-L  Studen 

Reporting  of  Instructor  Classroom  Facilitation  Behavior 

Facilitation  Behavior 

N  respondents 

%  Reporting  at  least  “more  did  than  didn’t” 

All 

Feb-start 

Aug-start 

Guidance 

250 

86% 

78% 

90% 

Feedback 

249 

79% 

71% 

83% 

Challenge 

247 

79% 

71% 

82% 

Group 

250 

79% 

68% 

84% 

Questions 

250 

89% 

81% 

92% 

As  shown  in  the  table  above,  a  substantial  proportion  of  students  reported  the  presence  of 
several  instructor  facilitation  behaviors  in  the  classroom.  Notably,  this  proportion  was  higher  for 
2006  August-start  students  on  all  behaviors,  suggesting  that  instructors  developed  as  facilitators 
of  adult  learning  between  the  two  cycles  of  the  course.  This  interpretation  should  be  viewed  with 
caution,  however,  because  August-start  students  had  only  recently  begun  classes  when  surveyed 
whereas  the  February-start  students  were  close  to  graduating.  The  data  collected  cannot  rule  out 
the  possibility  that  the  difference  shown  in  the  table  above  was  due  to  such  group  differences  as 
level  of  fatigue  or  habituation  to  the  course  and/or  reliance  on  short-term  versus  long-term 
memory  to  make  judgments. 

Summary.  A  very  high  proportion  of  surveyed  students  (79%  on  average)  reported  that 
ILE  instructors  generally  demonstrated  the  classroom  facilitation  behaviors  recognized  as  critical 
to  developing  adult  learners.  It  is  unknown  whether  this  proportion  would  be  the  same  for 
students  who  took  ILE  prior  to  the  launch  of  the  LVN  LLI.  However,  the  pattern  of  responding 
suggests  that  ILE  instructors  improved  their  facilitation  skills  between  the  2006  February-  and 
2006  August-start  course  cycles.  On  all  but  one  of  the  facilitation  behaviors  sampled,  a  greater 
proportion  of  August-start  students  reported  a  higher  frequency  of  instructor  facilitation. 

Although  alternative  explanations  for  this  pattern  cannot  be  ruled  out,  the  finding  could  indicate 
that  adoption  of  the  LVN  LLI  is  having  an  effect  on  instructional  quality.  A  more  in-depth 
evaluatuion  of  classroom  facilitation  behavior  involving  actual  classroom  observation  across 
course  cycles  is  warranted. 
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Outcome  -  Presence  of  Learning  Community 

The  emergence  of  a  community  of  learners  is  believed  to  be  both  an  outcome  of 
successful  blended  learning  and  a  key  determinant  of  lifelong  learning  orientation.  Lifelong 
learners  rely  on  the  support  of  the  learning  community  to  acquire  information  and  test  ideas, 
among  other  things  (Haythomthwaite,  2002;  Wellman  et  al.,  1988).  Seven  metrics  were 
developed  to  capture  learning  community,  but  only  one  was  measured  in  the  present  evaluation 
due  to  time  constraints  and  emphasis  on  other  metrics  of  more  immediate  interest  to  project 
stakeholders.  This  metric  was  selected  because  it  directly  addresses  the  use  of  a  LVN  LLI 
component  (SharePoint)  in  enabling  community  support  for  learners. 

1 .  Presence  (Yes/No)  of  a  student  leader  using  SharePoint  or  other  means  to 
collaboratively  organize  student  efforts  (due  date  lists,  class  calendar,  etc.).  Students  were 
surveyed  to  determine  whether  there  was  a  student  leader  in  their  staff  group  who  used 
SharePoint  to  coordinate  student  efforts.  Of  the  250  students  who  reported  on  student  leaders, 
57%  (77=  144)  indicated  that  their  staff  group  had  a  student  leader  (appointed  or  emergent)  who 
used  SharePoint  to  coordinate.  Thirty-six  percent  (77  =  89)  indicated  that  there  was  a  student 
leader,  but  that  he  or  she  did  not  use  SharePoint  to  coordinate.  The  proportion  of  student  leaders 
using  SharePoint  was  greater  for  August-start  students  than  for  February-start  students  [64% 

(101  of  159)  and  58%  (42  of  72),  respectively]. 

Outcome  -  Student  Responsibility  for  Own  Learning 

Four  metrics  were  selected  to  capture  the  degree  to  which  ILE  students  had  taken 
responsibility  for  their  own  learning  process.  Instruction  designed  to  facilitate  the  adult  learning 
process,  enabled  by  a  blended  learning  environment  with  easy  access  to  web-based  resources  and 
expert  communities,  should  lead  to  increased  resourcefulness  on  the  part  of  students  as  they 
begin  to  see  themselves  as  active  agents  in  their  own  learning  (McLoughlin  &  Oliver,  1999; 
Sherry  &  Wilson,  1997)  The  purpose  of  the  selected  metrics  was  to  explore  whether  students 
leveraged  the  LVN  LLI  to  conduct  independent  evaluation  in  ILE.  A  detailed  description  of  each 
metric  is  provided  below. 

1.  Percent  of  registered  students  who  used  Battle  Command  Knowledge  System  (BCKS) 
Leader  Network  to  supplement  course  materials,  complete  assignments,  or  for  own  research. 

This  metric  was  captured  by  asking  students  to  indicate  via  survey  how  important  BCKS  was  to 
them  when  searching  for  current  information  to  complete  class  assignments  or  conduct 
independent  research.  Students  were  asked  to  indicate  the  importance  of  the  BCKS  Leader 
Network  relative  to  other  KM  resources.  Seventy-five  students  (30%,  75  of  249)  reported  that 
they  did  not  search  for  current  information.  Of  the  remaining  1 74  students  who  responded  to  this 
survey  question,  30%  (77  =  52)  indicated  that  they  used  BCKS  as  much  or  more  than  other 
(Army  and  non-Army)  KM  resources. 

2.  Percent  of  registered  students  who  used  other  Army  KM  resources  (including  SMEs 
and  other  forums)  to  supplement  course  materials,  complete  assignments,  or  for  own  research. 
Similar  to  the  previous  metric,  students  were  asked  to  report  on  the  relative  importance  of  Army 
KM  resources  (excluding  BCKS,  e.g.,  AKO,  Center  for  Army  Lessons  Learned,  etc.)  when 
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completing  class  assignments  or  conducting  independent  research.  Twenty-seven  students  (11%, 
27  of  250)  reported  that  they  did  not  search  for  current  information.  Of  the  remaining  239 
students,  60%  ( N  =  144)  indicated  that  they  used  Army  KM  resources  as  much  or  more  than 
BCKS  or  non- Army  resources. 

3.  Percent  of  registered  students  who  used  non-Army  resources  to  supplement  course 
materials,  complete  assignments,  or  for  own  research.  As  above,  this  metric  was  captured  using 
a  single  survey  question  in  which  students  were  asked  to  indicate  the  importance  of  non- Amy 
KM  resources  relative  to  Army  KM  resources  (including  BCKS)  when  searching  for  current 
infonnation  to  conduct  academic  activity.  Five  students  (2%,  5  of  251)  reported  that  they  did  not 
search  for  current  information.  Of  the  remaining  246  students,  96%  (N=  236)  indicated  that  they 
used  non-Army  KM  resources  as  much  or  more  than  Army  KM  resources. 

4.  Percent  of  registered  students  who  accessed  a  particular  Blackboard  page  prior  to 
when  the  corresponding  class/lesson  was  held.  A  single  survey  question  was  used  to  capture  this 
metric.  Students  were  asked  to  report  how  frequently  (never,  sometimes,  often,  always,  and  N/A) 
they  accessed  course  content  in  Blackboard  prior  to  a  class  meeting  in  order  to  prepare.  Of  the 
249  students  who  responded  to  this  survey  question,  9%  ( N  =  23)  indicated  that  their  staff  group 
did  not  have  content  posted  on  Blackboard,  so  they  did  not  access  it  prior  to  class.  Of  the 
remaining  226  students,  55%  reported  that  they  often  or  always  used  access  to  Blackboard  to 
prepare  for  class. 

Summary.  A  fairly  high  proportion  of  students  (between  70%  and  98%)  reported  using 
knowledge  management  (KM)  resources  to  search  for  current  information  to  conduct  academic 
activity,  indicating  that  students  are  taking  charge  of  their  own  learning.  Most  students  who 
conducted  searches  (82%)  used  non-Army  resources  alone  or  more  than  Army  resources, 
however,  so  it  is  unclear  whether  the  LNV  LLI  in  specific  has  had  an  impact  on  student 
resourcefulness.  Fifty-five  percent  of  students  whose  course  materials  were  posted  in  Blackboard 
(9%  claimed  that  their  course  materials  were  not  posted  in  Blackboard)  reported  that  they 
accessed  the  LVN  LLI  prior  to  class  in  order  to  prepare.  This  finding  suggests  that  access  from 
outside  the  schoolhouse  is  being  used  to  support  student  performance  in  the  absence  of  printed 
materials  distributed  by  the  college. 

Outcome  -  Learning  Self-Efficacy 

Learning  self-efficacy  is  believed  to  be  an  important  outcome  of  any  learning  experience, 
regardless  of  whether  it  is  technology-supported  (e.g.,  Abell,  2003).  The  assumption  behind 
assessing  learning  self-efficacy  in  the  present  evaluation  was  that  high  levels  of  learning  self- 
efficacy  would  reflect  the  successful  application  of  technology  to  the  learning  environment.  In 
other  words,  they  would  reflect  that  the  introduction  of  technology  did  not  interfere  with  self- 
efficacy  and  that  the  technology  could  support  levels  of  self-efficacy  that  are  consistent  with  an 
external  criterion.  Learning  self-efficacy  has  only  one  corresponding  metric  in  the  metric  chart 
(see  Appendix  A),  which  was  assessed  in  the  present  evaluation. 

1 .  Percent  of  students  who  reported  that  they  were  capable  of  leading  their  own  learning 
process.  Students  were  asked  via  survey  to  report  how  capable  they  felt  of  leading  their  own 
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learning.  Students  had  the  option  of  indicating  whether  they  had  the  opportunity  to  lead  their 
own  learning  or  if  students  should  lead  their  own  learning.  Respondents  having  the  opinion  that 
students  should  be  able  to  lead  their  own  learning  had  the  option  of  indicating  how  well  they 
believed  they  accomplished  their  learning  goals.  Of  the  250  students  who  responded  to  this 
survey  question,  only  one  felt  that  students  should  not  lead  their  own  learning.  Of  the  remaining 
249  students,  88%  (N-  222)  indicated  that  they  had  accomplished  (February-start)  or  were 
accomplishing  (August-start)  most  or  all  of  their  learning  goals  during  ILE.  This  high  proportion 
of  students,  which  was  largely  the  same  for  the  February-  and  August-starts,  suggests  that  the 
instructional  technologies  comprising  the  LVN  LLI  are  not  preventing  high  levels  of  learning 
self-efficacy,  although  it  is  unknown  to  what  degree  the  technologies  may  help  foster  this 
orientation  or  to  what  degree  students  goals’  for  their  own  learning  match  those  of  the 
educational  institution. 

Outcome  -  Motivation  to  Learn 

As  with  learning  self-efficacy,  motivation  to  leam  is  a  critical  outcome  of  any  learning 
situation,  especially  for  adult  learners  for  whom  education  is  not  compulsory  (Abell,  2003). 
Technology  supported  learning  environments  should  not  reduce  motivation  to  leam,  and  may 
even  enhance  motivation,  if  the  technology  is  easy  to  access  and  well-designed.  Learning 
technology  also  may  affect  motivation  indirectly  through  increased  alignment  of  course  content 
with  operational  demands  (i.e.,  enhanced  course  relevance).  There  is  only  one  metric  in  the 
completed  metric  chart  that  is  associated  with  motivation  to  leam  (see  Appendix  A),  and  it  was 
assessed  in  the  present  evaluation. 

1 .  Percent  of  students  who  reported  that  they  were  motivated  to  learn  subjects  addressed 
by  the  courses  they  took.  Students  were  surveyed  to  assess  their  motivation  to  leam  the  subjects 
taught  in  ILE.  A  single  question  allowed  students  to  indicate  their  level  of  motivation  as  a 
function  of  content  relevance  and  instructor  attitudes.  Of  the  251  students  who  reported  their 
motivation  to  leam,  91%  (N  -  228)  indicated  that  they  felt  generally  (69%,  N—  172)  or  highly 
(22%,  N  =  56)  motivated  (i.e.,  that  course  content  spoke  to  their  interests  and  that  instructors 
were  engaging).  Although  it  is  impossible  to  tell  from  these  data  what  positive  impact  the  LVN 
LLI  has  on  ILE  students’  motivation,  it  is  clear  that  a  large  proportion  of  ILE  students  surveyed 
were  motivated  to  leam.  This  finding  suggests  that  a  negative  impact  probably  did  not  occur. 

Outcome  -  Enhanced  Instructional  Efficiency 

Of  the  three  metrics  for  instructional  efficiency  listed  in  the  metric  chart,  only  one  was 
selected  for  the  present  evaluation  due  to  time  limitations  and  the  irrelevance  of  one  of  the 
metrics  to  the  LVN  LLI.  The  selected  metric  was  chosen  over  the  other  relevant  metric  because 
it  more  directly  reflects  the  impact  of  the  educational  technology  introduced  via  the  LVN  LLI. 
Specifically,  the  LVN  LLI  may  enhance  classroom  efficiency  by  reducing  the  amount  of 
teaching  time  spent  doing  administrative  tasks.  Blackboard  enables  instructors  to  make 
announcements,  distribute  handouts,  collect  homework,  and  administer  exams,  among  other 
administrative  tasks,  all  outside  of  the  classroom  setting. 
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1 .  Estimated  reduction  in  time  spent  in  classroom  doing  administrative  tasks,  including 
announcements,  handouts,  and  testing.  In  a  single  survey  question,  instructors  were  asked  to 
estimate  how  much  classroom  time  spent  on  administrative  tasks  was  reduced  (relative  to  pre- 
LVN  LLI  situations)  by  using  Blackboard.  Of  the  69  instructors  who  responded  to  this  question, 
10%  (N-  7)  indicated  that  using  Blackboard  to  conduct  administrative  tasks  reduced  their 
classroom  administration  time  and  58%  (N=  40)  reported  no  change  in  classroom  administration 
time. 

Outcome  -  Enhanced  Organizational  Commitment 

Affective  organizational  commitment  is  a  component  of  individual  readiness  and 
perceptions  of  organizational  support  are  a  critical  determinant  of  this  commitment  that  can  be 
affected  directly  by  organizational  initiatives  (McGonigle  et  al.,  2005).  Of  the  five  metrics 
developed  to  capture  the  effect  of  Lifelong  Learning  Centers  on  perceptions  of  organizational 
support,  only  one  was  selected  for  measurement  in  order  to  manage  the  scope  of  the  survey 
administered  to  ILE  students  and  to  focus  the  evaluation  on  topics  of  immediate  interest  to 
stakeholders. 

1 .  Percent  of  users  who  reported  feeling  that  the  LVN  LLI  reflected  the  Army ’s  intent  to 
support  the  warfighter.  Currently  comprising  the  vast  majority  of  users  of  the  LVN  LLI,  ILE 
students  were  surveyed  to  assess  the  effect  of  the  LVN  LLI  on  perceptions  of  organizational 
support.  Students  were  asked  what  they  thought  to  be  the  main  reason  why  the  Army  decided  to 
administer  ILE  using  Blackboard  and  SharePoint.  Students  could  identify  financial,  political, 
educational,  unknown,  or  other  motives  for  the  change.  Student  suggestions  for  other  motives  for 
the  change  were  reviewed  for  their  alignment  with  the  given  motive  categories,  and  were  in  some 
cases  re-assigned  if  a  close  match  was  determined.  For  example,  responses  indicating  that  the 
reason  for  Blackboard  and  SharePoint  was  that  the  Army  wanted  to  advance  educational 
technology  were  recoded  as  educational  in  nature.  Similarly,  if  a  response  referred  to  making 
paper  projects  digital,  it  was  recoded  as  financial  in  nature.  Responses  suggesting  that  the  Army 
had  decided  to  use  Blackboard  and  SharePoint  to  keep  up  with  civilian  universities  or  to  move 
the  Anny  distance  learning  program  along  were  not  recoded. 

Of  the  251  students  who  reported  their  opinion  regarding  the  Anny’s  motives,  33% 
indicated  that  they  believed  the  Army’s  motives  were  educational  in  nature  and  in  the  best 
interests  of  leader  education.  Twenty- five  percent  of  February-start  students  (20  of  80)  selected 
educational  motives,  whereas  37%  of  August-start  students  (63  of  169)  selected  the  same.  This 
finding  suggests  that  the  LVN  LLI  currently  does  not  serve  to  reinforce  the  majority  of  students’ 
perceptions  organizational  support.  For  this  reason,  it  may  have  limited  impact  on  affective 
organizational  commitment.  The  proportion  of  students  for  which  the  LVN  LLI  may  influence 
perceptions  of  organizational  commitment  appears  to  be  increasing,  however,  alternative 
explanations  for  this  trend  (i.e.,  the  point  in  the  academic  program  at  which  students  were 
surveyed)  cannot  be  ruled  out. 
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Outcome  -  Cost-Effectiveness 

Of  the  six  cost-effectiveness  metrics  presented  in  the  metric  and  measures  chart,  one  was 
selected  for  the  present  evaluation— a  comparison  of  the  cost  per  student  to  deliver  ILE  using 
traditional  means  versus  the  LVN  LLI.  The  comparison  cases  used  to  build  this  metric  were  the 
first  full  pilot  year  of  the  LVN  LLI  (February  2006  to  present,  including  all  student  types)  and 
the  two  pilot  years  of  the  ILE  curriculum  change  from  the  traditional  CGSOC  (2003-2004  and 
2004-2005,  all  student  types). 

An  analysis  of  the  cost  per  student  of  a  blended  learning  initiative  must  reflect  the  costs 
associated  with  distributing  course  materials  as  well  as  those  costs  associated  with  producing  the 
curriculum  content  and  providing  instruction  and  course  administration  (e.g.,  enrollment,  record 
keeping,  etc.;  see  Cukier,  1997;  Levin,  1986).  This  approach  is  followed  regardless  of  the  means 
used  for  accessing  course  materials  (i.e.,  classroom,  video  teleconference,  web-based  content 
repositories,  etc.).  A  cost  accounting  approach  that  goes  beyond  instructional  technology  is 
necessary  because  a  complex  of  interrelated  expenses  comprises  the  total  cost  of  education, 
including  (a)  direct  personnel,  supplies,  and  other  costs,  including  printing  and  shipping  of 
course  materials;  (b)  indirect  support  costs,  such  as  facilities  and  maintenance,  administrative 
staff,  faculty  development,  etc.;  and  (c)  opportunity  costs.  Indeed,  the  cost  of  technology  to 
support  a  blended  learning  initiative  is  a  relatively  minor  expense  compared  to  the  other 
expenses  involved  in  education  (Levin,  1986). 

The  fiscal  promise  of  blended  learning— and  the  LVN  LLI  in  parti cular-is  that  it  will 
reduce  printing  and  shipping  costs  for  course  materials  (and  perhaps  eventually  travel  and 
housing  costs  for  students),  although  a  corresponding  increase  in  technology-related  costs  also  is 
expected.  In  order  to  understand  the  cost-effectiveness  of  the  LVN  LLI  in  the  context  of 
providing  ILE  to  all  Army  majors,  the  financial  tradeoff  involved  in  placing  course  materials  on 
the  web  must  be  considered  as  a  proportion  of  bigger  picture  expenses.  Presenting 
comprehensive  cost  data  not  only  allows  decision  makers  to  keep  educational  technology  costs  in 
perspective  but  also  allows  decision  makers  to  determine  what  future  efforts  should  have  the 
greatest  impact  on  the  cost  of  education. 

Cost  per  student  for  administering  ILE  in  the  LVN  LLI  vs.  using  traditional  means.  To 
develop  a  robust  cost  per  student  comparison  metric,  best  practices  in  cost  accounting  for  higher 
education  and  technology-supported  learning  were  applied  (Cukier,  1997;  Hyatt,  1983;  Levin, 
1986;  Middaugh,  2000;  Milam,  n.d.).  Briefly,  these  cost  accounting  methods  require  capturing 
the  direct  and  indirect  costs  allocated  to  education  delivery.  Direct  costs  include  personnel 
salaries  and  benefits,  supplies,  travel,  and  other  directly  allocable  expenditures,  such  as  contracts 
and  printing  and  shipping  costs.  Indirect  costs  are  broken  down  into  annual,  variable 
expenditures  (i.e.,  costs  associated  with  institutional  support,  such  as  facilities  maintenance  and 
faculty  and  staff  development)  and  annual  use  charges  associated  with  fixed  expenditures  (i.e., 
buildings  and  information  technology). 

Ideally,  all  costs,  no  matter  how  seemingly  trivial,  would  be  accounted  for  when 
developing  the  cost  per  student  metric.  There  was  limited  time  to  collect  financial  data  in  the 
present  effort  however,  so  decisions  were  made  to  manage  the  scope  of  the  work  while 
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producing  a  reasonable  reflection  of  (a)  change  in  cost  per  student  across  mode  of  content 
delivery;  and  (b)  change  in  cost  per  student  as  a  proportion  of  total  cost.  The  criteria  for 
including/excluding  cost  data  from  the  analysis  were  relevance  to  project  sponsors,  immediacy 
of  involvement  in  ILE  delivery,  presence  in  CGSC  financial  reporting,  and  data  availability. 

A  summary  of  inclusions  and  exclusions  is  presented  in  Table  B-2  below.  Direct  costs 
associated  with  staffing  and  supplying  the  TASS  Battalions  were  excluded  because  these  costs 
are  incurred  by  organizations  outside  of  the  CGSC  (e.g.,  the  Army  Training  Support  Center)  and 
because  reproduction  and  shipping  costs  lag  approximately  one  year  behind  the  implementation 
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ble  B-2.  Inclusions  and  Exclusions  in  the  Present  Cost  per  Student  Metric 

MH'iW'MfW1 

Item 

Included 

Excluded 

Direct  Costs 

Personnel 
(salaries,  benefits, 
etc.)  to  deliver 

ILE 

SADL  personnel  and  staff 
located  at  Fort  Leavenworth 

CGSS  faculty  (resident  and 
satellite)  and  admin  staff 
(requested  data  not  made 
available);  Faculty  and  admin 
staff  located  at  the  TASS 
Battalions 

Supplies 

CGSS  (resident  and  satellite) 
and  SADL  supplies 

Supplies  for  TASS  Battalions 

Other 

CGSS  printing  and  reproduction 
and  shipping  to  satellite 
locations;  DOET  contract  costs 
associated  directly  with  the  LVN 
LLI  (hardware,  software, 
supplies,  personnel,  and  travel) 

SADL  printing  and  reproduction 
and  shipping  to  TASS  Battalions 
and  individuals  (incurred  by  the 
Army  Training  Support  Center) 

Indirect 

Costs 

Institutional 

Support  -  CGSC 

DOET  and  QAO  direct  costs 
(personnel,  supplies,  and  other 
costs,  allocated  by  estimated 
percent  personnel  time  spent  on 
CGSS  and  ILE) 

FSDO  direct  costs;  CGSC 
executive  staff  (e.g., 
commandant)  and  all  other 
institutional  support  offices  in 
CGSC  (e.g.,  academic  outreach, 
library,  etc.) 

Institutional 

Support  -  Fort 
Leavenworth 

None 

All  (e.g.,  housing,  food  service, 
post  security,  etc.) 

Facilities 

Replacement  costs  for 
information  technology, 
classrooms,  offices,  etc.,  for 

CGSS  presence  in  Bell  Hall 
(represented  as  annual  use 
charges,  allocated  by  assignable 
square  feet  of  the  New  Lewis  & 
Clark  Building) 

Replacement  costs  for 
information  technology, 
classrooms,  offices,  etc.,  for 

CGSS  presence  in  Eisenhower 

Hall,  for  SADL  (located  in 
Eisenhower  Hall),  for  satellite 
locations,  and  for  TASS 

Battalions 

Opportunity 

Costs 

All  (i.e.,  cost  of 
activities  not 
conducted  because 

resources  were 
allocated  to  ILE) 

None 

All 
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of  a  new  initiative3.  Direct  costs  for  CGSS  personnel  were  excluded  because  the  requested  data 
were  not  made  available  for  investigation  and  in  any  case  would  have  spuriously  increased  the 
LVN  LLI  cost  per  student  metric  due  to  cost-of-living  raises.  Selected  indirect  costs  associated 
with  institutional  support  (within  the  CGSC  and  across  Fort  Leavenworth)  were  excluded 
because  they  were  not  of  primary  interest  to  the  project  sponsors  and  required  installation-wide 
coordination  efforts  that  exceeded  the  scope  of  this  short-term  research.  Indirect  facilities  costs 
associated  with  the  CGSC  presence  outside  of  Bell  Hall  (soon  to  be  moved  to  the  new  Lewis  & 
Clark  Building)— i.e.,  Eisenhower  Hall,  satellite  campus  facilities,  and  TASS  Battalion  facilities — 
were  excluded  because  replacement  cost  data  were  not  readily  available  and  required  significant 
coordination  beyond  the  scope  of  the  research.  Finally,  opportunity  costs  were  not  considered 
because  they  would  be  speculative  at  best. 

The  overall  impact  of  these  exclusions  on  the  cost  per  student  metric  is  that  (a)  cost  per 
student  for  both  traditional  and  LVN  LLI  delivery  will  be  significantly  underestimated;  and  (b) 
the  relative  change  in  cost  per  student  across  mode  of  content  delivery  will  be  significantly 
overestimated.  In  addition,  because  cost-of-living  escalation  could  not  be  accounted  for,  the 
change  in  cost  per  student  is  slightly  higher  than  it  would  be  if  such  escalation  was  addressed. 

The  Faculty  and  Staff  Development  Office  (FSDO)  was  considered  for  inclusion  as  an 
indirect  cost  because  it  supports  the  educational  effectiveness  of  the  LVN  LLI  through 
developing  and  certifying  ILE  instructors.  Changes  in  the  demand  for  instructors  or  in  the 
demand  for  specialized  instructor  skill  sets  could  have  an  impact  on  the  staffing  requirements 
and  personnel  allocation  of  the  FSDO.  Similarly,  the  Quality  Assurance  Office  (QAO)  was 
considered  as  an  indirect  cost  because  the  staffing  or  personnel  allocation  of  the  QAO  can  be 
affected  by  changes  introduced  by  the  LVN  LLI.  The  QAO  provides  annual  and  periodic 
educational  program  evaluations  for  the  CGSC,  and  the  introduction  of  new  technology  produces 
a  requirement  to  evaluate  that  technology,  perhaps  independently  of  the  course  to  which  the 
technology  has  been  applied.  FSDO  costs  were  excluded  from  the  present  cost  metric  because 
there  has  not  yet  been  a  change  in  FSDO  staffing  requirements  due  to  the  LVN  LLI.  FSDO 
personnel  expenses  over  the  past  several  years  associated  solely  with  cost  of  living  increases 
would  spuriously  increase  the  LVN  LLI  cost-per-student  relative  to  previous  conditions. 

The  widely  recognized  challenges  to  collecting  cost  per  student  metric  data  include  (a) 
differences  between  the  financial  reporting  methods  of  the  institution  and  the  financial  reporting 
requirements  of  cost  accounting;  (b)  determining  the  basis  for  allocating  indirect  costs;  (c) 
quantifying  opportunity  costs;  and  (d)  accounting  for  all  known  costs  associated  with  education 
delivery  (e.g.,  Middaugh,  2000).  Consistent  with  reports  of  other  program  evaluations  in  higher 
education,  all  of  these  challenges  were  encountered  in  the  present  evaluation.  The  cost  per 
student  comparison  metric  therefore  must  be  considered  an  estimate  serving  illustrative  purposes 
rather  than  a  hard  number  reflecting  actual  resource  expenditure. 


3  Discussion  with  staff  at  the  Army  Training  Support  Center  indicated  that  approximately  $200K  in  curriculum 
production  and  shipping  will  be  saved  by  placing  ILE  on  the  web.  The  initiation  of  these  savings  will  not  be 
observed  until  2008  and  will  not  be  sudden,  but  placing  ILE  on  the  web  eventually  will  result  in  an  annual  savings 


of  $200K. 
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Overall,  the  abbreviated  cost  per  student  for  delivering  ILE  using  the  LVN  LLI  relative  to 
the  abbreviated  cost  of  delivering  the  ILE  pilot  curriculum  using  traditional  means  is  greater  by  a 
factor  of  1.24.  It  cost  $163.14  more  per  student  in  the  first  year  to  deliver  ILE  using  the  LVN  LLI 
initiative  than  using  traditional  means  averaged  over  two  years.  This  increase  is  trivial  compared 
to  the  overall  cost  of  providing  ILE,  which  is  significantly  underestimated  due  to  the  excluded 
costs  described  above.  Had  these  costs  been  included,  the  relative  cost  of  delivering  ILE  via  the 
LVN  LLI  would  have  been  even  lower. 

Selected  Output  Metrics  -  Actual  System  Usage 

Three  metrics  capturing  the  actual  usage  of  the  LVN  LLI  were  measured  in  an  attempt  to 
understand  better  the  outcome  metrics  presented  above.  Several  of  these  outcomes,  such  as  high 
proportions  of  student  learning  self-efficacy,  self-leadership,  and  motivation,  cannot  be  linked 
definitively  to  the  LVN  LLI  as  a  causal  factor  because  the  analogous  data  were  not  collected 
prior  to  its  implementation.  In  other  words,  there  was  nothing  to  compare  the  present  outcome 
data  to,  so  it  is  unknown  whether  it  represents  a  change.  Usage  data  should  illuminate  the 
potential  impact  of  the  LVN  LLI  on  the  outcomes  measured  because  use  of  the  system  is 
necessary  to  achieve  impact.  Use  data  also  reveals  relative  demand  on  the  system,  which  may 
have  implications  for  technical  support  staffing  decisions  or  for  identifying  the  geographic 
location  of  most  users.  The  three  usage  metrics  described  below  were  selected  in  order  to  explore 
whether  or  not  the  educational  technology  made  available  to  curriculum  developers,  instructors, 
and  students  was  used  to  enhance  the  learning  experience.  A  detailed  description  of  each  metric 
is  provided  below. 

1.  Pattern  of  access  as  a  function  of  time  of  day.  Because  ILE  course  content  is  stored  in 
Blackboard,  and  student  access  to  Blackboard  can  reflect  direct  interaction  with  curriculum 
materials,  patterns  in  student  hits  to  the  Blackboard  document  repository  were  examined. 
Specifically,  the  shape  of  the  frequency  distribution  of  hits  as  a  function  of  time  of  day  (hour  by 
hour)  was  analyzed.  This  analysis  was  conducted  using  data  collected  automatically  by 
Blackboard  for  resident  and  satellite  ILE  students. 

Between  June  20  and  September  21,  2006,  Blackboard  had  received  48,652  hits.  The 
Blackboard  data  received  to  conduct  further  analysis  provided  only  a  summary  of  the  frequency 
data  (i.e.,  total  number  of  hits,  a  histogram  plot  of  the  hits  as  a  function  of  time  of  day,  and  a 
listing  of  the  number  of  hits  per  time  of  day  and  their  corresponding  percentage),  so  a 
representative  sample  of  these  hits  (487,  or  1%,  preserving  the  relative  frequency  of  hits  as  a 
function  of  time  of  day)  was  used  to  plot  the  distribution  of  hits  and  test  the  normality  of  this 
distribution,  as  shown  in  Figure  B-3. 

A  Kolmogorov-Smimov  test  of  normality  was  applied  to  the  sample  dataset,  and  was 
significant  (p  =  .044),  suggesting  that  the  distribution  of  hits  a  function  of  time  of  day  is  non¬ 
normal.  The  Kolmogorov-Smimov  test  is  known  to  be  sensitive  to  large  sample  sizes,  however, 
and  other  data  (i.e.,  skewness  =  -.019;  kurtosis  =  -.335)  indicate  that  the  deviation  from 
normality  is  slight.  Visual  inspection  of  the  data  reveals  that  students  primarily  access  the 


4  The  exact  estimates  and  calculations  used  to  derive  this  metric  can  be  obtained  by  contacting  the  report  author,  Dr. 
Anna  T.  Cianciolo  (acianciolo@cpresearch.net). 
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Blackboard  document  repository  during  normal  business  hours,  with  reduced  access  at  other 
times,  but  that  there  is  a  spike  in  activity  after  the  dinner  hour,  between  7  and  10pm. 


Together,  these  findings  represent  an  expected  pattern  of  access  given  that  the  students 
accessing  the  Blackboard  content  repository  (i.e.,  students  in  residence  at  Fort  Leavenworth  and 
at  satellite  locations)  are  in  adjacent  time  zones  (i.e.,  Central  and  Eastern).  The  spike  in  access 
after  the  dinner  hour  suggests  that  students  are  using  the  system  to  conduct  educational  tasks 
while  at  home  or  otherwise  away  from  the  classroom  setting.  A  more  horizontal  distribution 
should  be  expected  in  a  complete  sample  of  user  activity  that  captures  reach  back  from  widely 
distributed  locations,  such  as  Hawaii,  Korea,  Europe,  and  the  Middle  East. 


Mean  =  13.7 
Std.  Dev.  =  4.784 
N  =  487 


Figure  B-3.  Pattern  of  Access  to  Blackboard  Content  Repository  as  a  Function  of  Time  of  Day 


2.  Frequency  of  access  to  system  during  normal  business  hours  vs.  off-hours.  The  same 
data  were  used  to  compute  this  metric  as  were  used  to  compute  the  previous  metric.  Business 
hours  were  defined  as  7am  to  5pm,  Central  Standard  Time,  to  account  for  classes  beginning  at 
8  am  in  the  Eastern  time  zone.  Approximately  69%  of  student  access  occurred  between  7am  and 
5pm,  and  31%  between  5pm  and  7am.  Of  the  off-hours  hits,  approximately  91%  occurred 
between  6am  and  7am  and  between  5pm  and  midnight.  Again,  this  pattern  is  expected  given  the 
student  population  using  the  system.  A  greater  equivalence  between  the  percentages  of  on-  and 
off-hours  hits  would  be  more  likely  if  the  hits  of  all  types  of  users  (i.e.,  graduates  using 
Blackboard  for  reachback)  were  tracked. 
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3.  Percent  of  registered  users  who  actively  used  the  system.  This  metric  was  captured  by 
combining  selected  survey  responses  given  by  curriculum  developers,  instructors,  and  students. 
Specifically,  a  set  of  four  yoked  questions  was  present  in  the  surveys  administered  to  the  three 
user  types  (one  question  each  in  the  curriculum  developer  and  instructor  surveys,  two  questions 
in  the  student  survey),  and  the  responses  to  these  questions  were  examined  collectively.  In  their 
respective  survey,  each  type  of  user  responded  to  a  question  in  which  they  were  asked  the 
primary  means  by  which  they  carried  out  educational  tasks.  For  example,  curriculum  developers 
were  asked  what  means  they  used  most  often  to  share  course  content  during  the  authoring 
process.  Instructors  were  asked  the  means  they  used  most  often  to  deliver  supplemental  course 
content.  Students  were  asked  the  primary  means  by  which  they  accessed  course  content  and 
coordinated  with  other  students5. 

Respondents  indicating  that  they  used  a  means  other  than  the  LVN  LLI  components 
“most  often”  or  “primarily”  to  conduct  educational  activities  were  not  considered  active  users  of 
the  system.  In  sum,  forty  percent  of  course  authors  (4  of  10)  used  SharePoint  to  collaborate  on 
curriculum  development,  30%  of  instructors  (21  of  69)  used  Blackboard  to  distribute  course 
materials,  37%  students  (92  of  251)  accessed  Blackboard  to  retrieve  course  materials,  and  5%  of 
students  (13  of  251)  used  SharePoint  to  coordinate  with  one  another  on  group  projects  or  other 
activities.  For  instructors  (N-  69),  a  greater  proportion  actively  used  Blackboard  to  teach 
August-start  than  to  teach  February- start  ILE  (34%  and  24%,  respectively).  In  contrast,  a  smaller 
proportion  of  August-start  students  reported  actively  using  Blackboard  than  February-start 
students  (28%  and  55%,  respectively).  The  reason  for  the  inconsistent  pattern  of  these 
differences  is  unknown.  The  relatively  small  proportion  of  students  using  SharePoint  to 
coordinate  with  one  another  likely  is  due  to  the  fact  that  (a)  the  students  surveyed  were  co¬ 
located  and  preferred  face-to-face  means  of  collaboration  (38%);  and  (b)  login  requirements 
made  using  SharePoint  from  home  more  difficult  to  use  than  email  (50%  reported  using  email, 
and  survey  responses  indicated  that  login  requirements  were  a  detractor). 

Summary.  System  (i.e.,  Blackboard  knowledge  repository)  access  data  indicate  that  the 
LVN  LLI  is  available  and  used  around  the  clock,  with  access  patterns  reflecting  those  expected  if 
users  are  students  primarily  located  within  the  same  or  adjacent  time  zones  and  if  these  student 
users  access  the  system  from  home  to  conduct  their  studies.  Peak  access  activity  occurred  during 
business  hours  and  in  the  early  morning  and  immediate  post-dinner  hours. 

In  the  schoolhouse,  the  LVN  LLI  components  (i.e.,  Blackboard  and  SharePoint)  appear  to 
be  competing  with  other  alternatives,  and  even  with  each  other,  for  usage.  That  is,  only  30%  of 
surveyed  instructors  reported  using  Blackboard  as  their  primary  means  for  distributing  course 
content,  with  SharePoint  being  the  most  frequently  reported  alternative  (39%  of  instructors). 
Students  surveyed  reasonably  reported  a  similar  pattern  of  use;  37%  reported  using  Blackboard 
most  often  to  access  course  content,  citing  SharePoint  most  frequently  as  the  alternative  (33%). 
Only  5%  of  surveyed  students  reported  using  SharePoint  to  collaborate  with  other  students,  with 
email  and  face-to-face  meetings  most  frequently  cited  as  alternatives  (50%  and  38%, 
respectively).  Comments  on  student  and  instructor  surveys  indicated  that  there  was  some 


5  It  was  assumed  that  100%  of  students  used  Blackboard  to  access  main,  or  standardized,  curriculum  content  because 
it  was  not  available  using  other  means. 
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confusion  as  to  why  there  were  two  systems  which  have  so  many  overlapping  features.  When 
forced  to  make  a  choice,  people  appear  to  use  the  most  readily  available  or  familiar  means  to 
conduct  their  academic  tasks. 

Selected  Activity  Metrics  -  Technical  Staff 

Four  metrics  capturing  the  activity  of  technical  staff  were  measured.  These  metrics  were 
selected  in  order  to  explore  the  degree  to  which  technical  staff  activity  could  be  expected  to 
support  curriculum  developer  and  instructor  use  of  the  software  comprising  the  LVN  LLI  (see 
DTIC,  2000;  Willis,  2000).  According  to  the  logic  model  developed  as  part  of  this  research, 
effective  use  of  the  software  is  a  critical  determinant  of  the  learning  effectiveness  of  the  system. 
A  detailed  description  of  each  metric  is  provided  below. 

1.  Percent  of  instructors  and  curriculum  developers  trained  to  technical  criterion.  The 
original  plan  for  capturing  this  metric  was  to  (a)  audit  Blackboard  and  SharePoint  instruction;  (b) 
develop  a  knowledge  or  performance  measure  to  capture  student  achievement  of  the  learning 
objectives;  and  (c)  determine  the  percentage  of  students  performing  successfully  on  the  special 
purpose  measure.  After  auditing  the  Blackboard  and  SharePoint  instruction,  however,  it  was 
determined  that  a  knowledge  or  performance  measure  would  not  be  necessary  to  capture  student 
achievement  and  that  all  students  achieve  the  technical  learning  objectives  upon  completion  of 
the  instruction. 

On-site  contractors  in  the  CGSC  DOET  provide  basic  Blackboard  and  SharePoint 
instruction  on  every  working  Friday  and  more  advanced  instruction  on  an  as-needed  basis  for 
individuals  or  groups  with  special  requests.  DOET  began  delivering  Blackboard  and  SharePoint 
instruction  in  July  of  2006,  prior  to  which  the  instruction  was  conducted  by  the  CGSC  FSDO. 
DOET  assumed  responsibility  for  conducting  the  training  so  that  technical  instruction  could  be 
delivered  by  staff  dedicated  to  understanding  and  maintaining  currency  with  the  technology. 
Participants  in  the  instruction  include  curriculum  developers  (i.e.,  block  and  lesson  authors), 
instructors,  and  ILE  students  (including  international  students)  who  have  voluntarily  sought  it 
out.  Basic  SharePoint  instruction  is  required  for  ILE  students  who  are  appointed  to  serve  as  the 
information  management  officers  for  their  staff  groups.  Learner  groups  in  the  Blackboard  and 
SharePoint  instruction  are  usually  small,  averaging  approximately  four  students,  and  most  of  the 
instruction  is  conducted  on  a  one-on-one  basis. 

Both  Blackboard  and  SharePoint  basic  instruction  are  accomplished  via  a  combination  of 
PowerPoint  presentation  and  practical  exercises.  Class  time  usually  lasts  approximately  one- 
hour.  The  Blackboard  instruction  has  three  technical  learning  objectives:  (1)  Add  an 
announcement;  (2)  Add  course  content;  and  (3)  Add  users  to  a  staff  group.  The  SharePoint 
instruction  has  five  technical  learning  objectives:  (1)  Create  a  library  (document  or  picture);  (2) 
Add  a  survey;  (3)  Add  a  custom  list;  (4)  Create  a  view;  and  (5)  Set  permissions.  Students  do  not 
leave  the  class  until  they  have  accomplished  these  learning  objectives. 

Two  survey  questions  were  developed  to  address  participation  in  and  retention  of  the 
Blackboard  and  SharePoint  instruction.  Both  curriculum  developers  and  instructors  were  asked 
whether  they  had  participated  in  Blackboard  or  Sharepoint  instruction  and  whether  they  had  to 
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relearn  anything  they  had  been  taught  during  the  instruction  in  order  to  perform  their  duties.  Of  9 
course  authors  and  67  instructors  who  indicated  that  they  had  taken  Blackboard  or  SharePoint 
instruction,  53%  (37  of  67  instructors;  3  of  9  course  authors)  reported  having  to  relearn  the 
instructional  content. 

It  is  unknown  which  of  these  respondents  participated  in  instruction  provided  by  the 
DOET  versus  the  FSDO.  Because  all  but  one  of  the  10  course  authors  surveyed  had  authored  for 
both  February-  and  August-start  ILE,  it  is  unlikely  that  they  participated  in  DOET  SharePoint 
instruction.  However,  among  the  instructors,  76%  (N=  13)  of  those  who  taught  only  the 
February-start  reported  relearning  Blackboard  instruction  whereas  only  44%  (N  =  17)  of  the 
August-start  reported  relearning.  This  trend  suggests  that  instruction  provided  by  the  DOET, 
which  began  in  July  of  2006,  may  be  producing  greater  retention. 

2.  Percent  of  instructors  and  curriculum  developers  trained  to  criterion  on  content 
management/delivery  procedures.  Interviews  with  the  DOET  technical  support  as  well  as 
curriculum  developers  and  instructors  indicated  that  formal  information-management  procedures 
for  Blackboard  or  SharePoint  do  not  exist.  Instead,  some  control  measures  were  built  into  the 
design  of  the  software  applications  by  the  DOET  technical  staff.  Beyond  these  control  measures, 
curriculum  developers  and  instructors  were  given  the  authority  to  create  informal  information- 
management  procedures  in  collaboration  with  their  working  groups  (and  with  the  DOET 
technical  staff,  if  desired).  For  these  reasons,  instructors  and  curriculum  developers  do  not 
receive  training  on  content  management/delivery  procedures. 

3.  Accessibility,  usability,  and  utility  of  help  documentation  for  instructor  and  curriculum 
developer  self-development.  The  accessibility,  usability  (i.e.,  readability),  and  utility  of  help 
documentation  was  assessed  using  three  separate  survey  questions  in  which  curriculum 
developers  and  instructors  reported  their  perceptions.  Instructors  reported  on  the  accessibility, 
usability,  and  utility  of  Blackboard  help  documentation,  which  includes  an  electronic  version  of 
the  Blackboard  user  manual  and  additional  online  help  for  basic  functions  developed  in-house  by 
the  DOET.  Curriculum  developers  reported  on  the  accessibility,  usability,  and  utility  of 
SharePoint  help  documentation,  which  includes  an  electronic  version  of  the  SharePoint  user 
manual. 

Of  the  69  instructors  who  responded  to  the  survey,  66  reported  on  the  accessibility  and 
usefulness  of  the  Blackboard  help  documentation,  and  68  reported  on  its  readability.  A 
substantial  proportion  of  respondents  for  each  question  indicated  that  they  had  never  looked  for 
or  used  the  help  documentation  (44%,  51%,  and  57%,  respectively).  Of  those  who  looked  for 
help  documentation,  58%  (21  of  36)  reported  that  they  could  usually  or  always  find  the  topics 
they  were  looking  for.  Thirty-nine  percent  of  respondents  (12  of  31)  who  used  the  Blackboard 
documentation  found  that  it  either  partially  or  completely  met  their  information  needs,  and  41% 
of  respondents  (12  of  29)  reported  that  the  Blackboard  documentation  was  understandable. 

Of  the  10  curriculum  developers  who  responded  to  the  survey,  10  reported  on  the 
accessibility  of  the  SharePoint  help  documentation,  9  on  its  usefulness,  and  8  on  its  readability. 
Most  course  authors  reported  either  having  not  looked  or  not  used  the  documentation.  The  single 
course  author  who  reported  looking  for  documentation  reported  usually  being  able  to  find  it.  Of 
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the  3  course  authors  who  reported  using  the  SharePoint  help  documentation,  none  of  them  found 
it  useful.  The  one  person  who  reported  on  the  readability  of  the  help  documentation  (7  of  8 
reported  not  using  it)  indicated  that  they  generally  had  no  trouble  with  readability.  It  is  unknown 
why  more  course  authors  reported  having  used  the  help  documentation  than  having  looked  for  it. 
It  also  is  unknown  why  the  number  of  authors  reporting  on  the  utility  of  the  help  documentation 
was  higher  than  the  number  reporting  having  used  it. 

4.  Perceived  accessibility/responsiveness  and  utility  of  technical  support.  Students, 
instructors,  and  course  authors  were  surveyed  regarding  their  perceptions  of  the  accessibility  and 
the  utility  of  the  technical  support  provided  by  the  DOET.  The  responses  of  these  different  users 
are  summarized  in  Table  B-3  below. 


Table  B-3.  Perceived  Accessibility  and  Utility  of  Technical  Support  -  Survey  Responses 


User  Type 

%  Looked  for  TS 

%  Accessible 

%  Used  TS 

%  Good  Experience 

Course 

Author 

80%  (8  of  10) 

88%  (7  of  8) 

80%  (8  of  10) 

88%  (7  of  8) 

Instructor 

71%  (49  of  69) 

69%  (34  of  49) 

64%  (44  of  69) 

73%  (32  of  44) 

Student 

71%  (178  of  250) 

58%  (104  of  178) 

63%  (156  of  247) 

63%  (98  of  156) 

Average 

71%  (235  of  329) 

62%  (145  of  23 5) 

64%  (208  of  326) 

66%  (137  of  208) 

Note,  TS  =  Technical  Support. 


As  shown  in  Table  B-3,  the  majority  of  course  authors,  instructors,  and  students  sought 
and  used  the  technical  support  provided  by  the  DOET.  Importantly,  the  majority  of  people  found 
the  technical  support  they  were  looking  for.  The  proportion  of  satisfied  users  was  larger  for 
August-start  students  and  instructors  than  for  February-start  students  and  instructors  (81%  good 
experience  versus  67%  good  experience  for  instructors,  and  67%  good  experience  versus  54% 
good  experience  for  students.)  All  but  one  of  the  course  authors  had  written  content  for  both 
February-  and  August-start  ILE. 

Summary.  The  technical  staff  appears  to  provide  adequate,  effective  support  to  LVN  LLI 
users.  It  provides  basic  training  on  the  functions  of  the  LVN  LLI  components  in  which  the  large 
majority  of  instructors  and  course  authors  surveyed  (>90%)  took  part.  A  minority  of  course 
authors  (33%)  and  August-start  instructors  surveyed  (44%)  reported  that  they  had  to  relearn 
some  of  the  instructional  material,  but  it  is  unclear  how  many  of  these  people  went  through  the 
training  provided  by  the  DOET  versus  the  FSDO.  Although  formal,  institutional  SOPs  for 
information  management  in  the  LVN  LLI  do  not  exist,  the  DOET  technical  staff  assists  teaching 
teams  and  curriculum  development  teams  in  creating  their  own  procedures.  Overall,  students, 
instructors,  and  curriculum  developers  reported  having  a  good  experience  with  finding  and  using 
technical  support  (62%  and  66%,  respectively).  The  reported  frequency  of  good  experiences  was 
higher  for  August-start  survey  participants  (70%)  than  for  February-start  participants  (56%).  In 
contrast,  the  Blackboard  and  SharePoint  help  documentation  appeared  to  be  of  limited  utility. 
The  small  majority  of  instructors  (58%)  could  find  Blackboard  help  documentation,  but  only 
39%  found  it  useful  or  readable  (41%).  The  large  majority  of  course  authors  surveyed  did  not 
look  for  or  use  SharePoint  help  documentation. 

Selected  Activity  Metrics  -  Curriculum  Developers 
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Ten  metrics  capturing  the  activity  of  curriculum  developers  (i.e.,  block  authors  and  lesson 
authors)  were  measured.  These  metrics  were  selected  in  order  to  explore  the  degree  to  which 
curriculum  developers  were  actually  using  SharePoint  to  collaborate.  These  metrics  also  were 
designed  to  capture  how  frequently  course  authors  were  leveraging  resources  made  available  in 
the  LVN  LLI  to  enhance  course  content.  According  to  the  logic  model,  effective  use  of 
SharePoint  and  other  LVN  LLI  resources  by  curriculum  developers  is  an  important  determinant 
of  the  efficiency  and  relevance  of  1LE  course  content.  A  detailed  description  of  each  metric  is 
provided  below. 

1.  Perceived  ease  of  using  SharePoint  to  share  documents  and  other  materials  used  in 
generating  course  content.  A  single  survey  question  was  used  to  assess  curriculum  developers’ 
perceptions  of  the  ease  of  using  SharePoint.  Of  the  9  course  authors  who  responded  to  this 
survey  question,  8  (89%)  indicated  that  it  was  easy  or  fairly  easy  to  post  and  locate  curriculum 
content  in  SharePoint. 

2.  Percent  of  curriculum  developers  who  reported  being  able  to  use  SharePoint  outside  of 
the  schoolhouse  setting.  A  single  survey  question  was  used  to  assess  curriculum  developers’ 
experiences  accessing  and  using  SharePoint  from  home  or  some  other  location  other  than  their 
office  at  Fort  Leavenworth.  Of  the  7  course  authors  who  responded  to  this  survey  question,  5 
(71%)  reported  having  no  real  problems. 

3.  Percent  of  curriculum  developers  using  SharePoint  to  store,  share,  and  revise  course 
content  materials.  Curriculum  developers’  use  of  SharePoint  to  store,  share,  and  revise 
curriculum  materials  was  captured  using  a  survey  question  in  which  respondents  selected  the 
primary  means  by  which  they  distributed  course  content  to  other  developers.  Respondents 
selecting  an  option  other  than  SharePoint  were  not  considered  to  be  using  SharePoint  to  store, 
share,  and  revise  because  they  were  actively  involved  in  using  some  other  means.  Of  the  1 0 
curriculum  developers  who  responded  to  the  survey,  4  (40%)  reported  using  SharePoint  as  the 
primary  means  for  distributing  course  content  materials  to  others. 

4.  Percent  of  curriculum  developers  who  reported  using  means  to  collaborate  other 
than/in  addition  to  SharePoint  (e.g.,  email,  thumb  drives,  etc.).  The  percentage  of  curriculum 
developers  using  means  in  addition  to/other  than  SharePoint  to  collaborate  was  assessed  with  the 
same  survey  question  described  above.  Of  the  10  curriculum  developers  who  responded  to  the 
survey  question,  6  of  10  (60%)  reported  using  something  other  than  SharePoint  as  the  primary 
means  for  distributing  course  content  materials  to  others.  The  most  frequently  cited  alternatives 
were  the  internal  network  (2  of  10)  and  email  (3  of  10). 

5.  Percent  of  curriculum  developers  using  Battle  Command  Knowledge  System  (BCKS) 
Leader  Network  to  enhance  course  content.  Curriculum  developers  were  asked  in  a  single  survey 
question  how  important  the  discussion  forums  in  the  BCKS  Leader  Network  were  for  them  when 
conducting  searches  to  enhance  course  content.  Respondents  answering  that  the  Leader  Network 
was  equally  or  more  important  relative  to  other  resources  were  assumed  to  be  using  actively  the 
Leader  Network.  Of  the  7  course  authors  who  reported  searching  for  current  information,  all  of 
them  reported  actively  using  the  Leader  Network  for  conducting  knowledge  searches. 
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6.  Percent  of  curriculum  developers  using  other  Army  Knowledge  Management  (KM) 
resources  to  enhance  course  content.  In  the  same  survey  question  described  above,  curriculum 
developers  were  asked  how  important  other  Army  KM  resources,  such  as  Army  Knowledge 
Online  or  the  Center  for  Army  Lessons  Learned,  were  for  them  when  conducting  knowledge 
searches.  Respondents  answering  that  Army  KM  resources  other  than  BCKS  were  equally  or 
more  important  relative  to  other  resources  were  considered  to  be  using  actively  Army  KM 
resources.  Of  the  9  course  authors  who  reported  searching  for  current  information,  7  (78%) 
reported  actively  using  Army  KM  resources  for  conducting  knowledge  searches. 

7.  Percent  of  curriculum  developers  using  non-Army  resources  to  enhance  course 
content.  The  same  survey  question  to  assess  the  above  two  metrics  was  used  to  assess  this  metric. 
Curriculum  developers  were  asked  to  indicate  how  important  non-Army  resources,  such  as 
Google  or  Early  Bird,  were  for  them  when  conducting  knowledge  searches  to  enhance  their 
course  content.  Of  the  1 0  course  authors  who  reported  searching  for  current  information,  2  (20%) 
reported  actively  using  Army  KM  resources  for  conducting  knowledge  searches. 

8.  Presence  of  formal  and/or  informal  standard  operating  procedures  (SOPs)  for 
leveraging  the  capabilities  of  the  software  applications,  including  content  organization  (Yes/No). 
The  presence  of  formal  and/or  informal  standard  operating  procedures  was  assessed  by 
interviewing  the  technical  support  staff  and  surveying  and  interviewing  curriculum  developers. 
The  interviews  indicated  that  SOPs  did  not  exist  formally  outside  of  the  structure  of  the 
SharePoint  site  for  curriculum  developers.  Informal  SOP  development  sometimes  was 
accomplished  when  working  groups  of  course  authors  requested  assistance  from  the  technical 
staff  in  leveraging  SharePoint’s  advanced  capabilities.  When  surveyed  about  their  participation 
in  SOP  development,  7  of  9  course  authors  responded  that  there  were  no  SOPs. 

9.  Percent  of  early  adopters  involved  in  SOP  development.  Curriculum  developers  were 
surveyed  to  assess  how  many  were  involved  in  SOP  development.  Early  adopters  were  not 
distinguished  from  late  adopters  in  the  present  research  because  all  curriculum  developers  using 
SharePoint  in  the  pilot  phase  and  assisting  in  SOP  development  were  considered  early  adopters. 
Of  the  9  curriculum  developers  who  reported  on  SOP  development,  2  reported  that  they  had  been 
involved  in  such  development  but  that  the  SOPs  they  worked  on  did  not  cover  everything  and 
were  not  always  followed. 

10.  Presence  of  a  formal  and/or  informal  mentorship  system  for  assisting  late  adopters 
leverage  software  capabilities  (Yes/No).  The  presence  of  a  formal  and/or  informal  mentorship 
system  was  assessed  by  interviewing  the  technical  support  staff  and  surveying  curriculum 
developers.  Both  sources  of  information  indicated  that  there  was  not  a  formal  mentorship 
program.  Informal  mentorship  in  the  form  of  advanced  SharePoint  and  Blackboard  classes,  was 
provided  by  the  DOET  technical  support  staff  on  an  as-needed  basis.  Individual  developers  or 
working  groups  of  developers  could  request  advanced  training  to  meet  their  specific 
requirements.  Course  authors  who  reported  on  mentoring  via  survey  indicated  that  help  with 
SharePoint  by  peers  was  largely  informal  (4  of  10  respondents  reported  asking  others  for  help) 
but  that  occasionally  more  formal  mentoring  occurred  (3  of  10  respondents  reported  having 
group  discussions  or  regularly  helping  others). 
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Summary .  Although  the  response  rate  for  course  author  surveys  was  67%,  it  must  be 
noted  that  only  1 5  course  authors  were  invited  to  participate  in  the  survey,  so  several  of  the 
curriculum  developer  metrics  are  based  on  the  reports  of  only  10  people.  That  said,  the  active 
adoption  rate  of  SharePoint  was  40%  among  course  authors.  Course  authors  do  not  appear  to 
have  had  much  difficulty  using  or  accessing  SharePoint.  Eighty-nine  percent  felt  SharePoint  was 
easy  or  fairly  easy  to  use  and  71%  reported  no  significant  problems  accessing  SharePoint  from 
home.  Instead,  adoption  rates  lower  than  criterion  (i.e.,  100%)  appear  to  be  linked  to  the  ready 
availability  of  flexible  alternatives  for  sharing  documents,  such  as  the  college  internal  network 
and  email. 

Course  authors  occasionally  helped  one  another  to  use  SharePoint,  but  there  does  not 
appear  to  be  a  consistent  formal  codification  of  what  was  learned.  Most  course  authors  stated  that 
there  were  no  SOPs  for  posting  content  in  SharePoint,  although  2  of  9  claimed  to  have  been 
involved  in  the  development  of  SOPs.  The  most  common  mode  of  seeking  help  was  to  ask  a 
peer,  rather  than  to  develop  a  collective  knowledge  base. 

Army  KM  resources  appear  to  have  been  most  important  to  curriculum  developers  when 
authoring  course  content.  The  large  majority  of  course  authors  stated  that  BCKS  Leader  Network 
(7  of  7)  and  other  Army  KM  resources  (7  of  9)  were  equally  or  more  important  than  non-Army 
resources,  such  as  Google  and  Early  Bird.  This  finding  suggests  that  curriculum  developers  are 
making  use  of  the  available  connection  to  the  operational  Army,  despite  the  fact  that  they  cannot 
accelerate  the  curriculum  production  process  in  order  to  respond  adaptively  to  what  they  find. 

Selected  Activity  Metrics  -  Instructors 

Eleven  metrics  capturing  the  activity  of  instructors  were  measured.  These  metrics  were 
selected  in  order  to  explore  the  degree  to  which  instructors  were  actually  using  Blackboard  to 
distribute  course-related  materials.  These  metrics  also  were  designed  to  capture  how  frequently 
instructors  were  leveraging  resources  made  available  in  the  LVN  LLI  to  supplement  the  formal 
curriculum.  According  to  the  logic  model,  effective  use  of  Blackboard  and  other  LVN  LLI 
resources  by  instructors  is  an  important  determinant  of  the  relevance  of  ILE  course  content 
(Clark,  1994;  Ehrmann,  1994;  Owston,  1997).  A  detailed  description  of  each  metric  is  provided 
below. 


1.  Percent  of  instructors  who  used  means  other  than  Blackboard  to  deliver  course 
content.  This  metric  was  assessed  with  a  single  survey  question  in  which  instructors  were  asked 
to  report  the  means  they  used  most  often  to  post  course  content.  Instructors  were  directed  to  post 
all  course  content  to  students  via  Blackboard,  so  a  selection  other  than  Blackboard  in  response  to 
this  question  was  intended  both  to  reflect  system  adoption  and  to  identify  the  preferred 
alternatives  to  system  use.  Of  the  69  instructors  who  responded  to  this  survey  question,  70% 
reported  using  something  other  than  Blackboard  most  often  when  posting  course  content.  The 
most  frequently  sited  alternative  means  for  posting  content  was  SharePoint  (39%,  27  of  69). 
Other  alternatives  sited  included  printed  materials  (17%),  email  (9%),  and  the  internal  network 
(4%). 
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Of  the  17  respondents  who  taught  only  February- start  ILE,  76%  (13  of  17)  reported  using 
means  other  than  Blackboard  to  post  content  most  often,  whereas  66%  (27  of  41)  of  the  2006 
August-start  respondents  reported  the  same.  This  trend  suggests  increasing  adoption  of 
Blackboard,  but  that  adoption  levels  remain  somewhat  low.  For  both  sets  of  respondents, 
SharePoint  was  the  most  frequently  cited  alternative  to  posting  in  Blackboard  (53%  and  37%, 
respectively). 

2.  Percent  of  instructors  who  used  BCKS  Leader  Network  to  post  supplemental  course 
content  in  Blackboard.  Instructors  were  asked  in  a  single  survey  question  how  frequently  they 
accessed  discussion  forums  in  the  BCKS  Leader  Network  to  conduct  information  searches  to 
enhance  course  content.  Respondents  answering  that  they  accessed  the  Leader  Network  daily  or 
weekly  were  considered  to  be  using  actively  the  Leader  Network.  Of  the  65  instructors  who 
responded  to  this  question,  12%  (N =  8)  responded  that  they  did  not  search  for  supplemental 
information.  Of  the  remaining  57  respondents,  4%  (N  ~  2)  reported  actively  (weekly  only)  using 
the  Leader  Network  for  conducting  knowledge  searches.  Seventy  percent  reported  never  using  it. 

3.  Percent  of  instructors  who  used  other  Army  KM  resources  to  post  supplemental  course 
content  into  Blackboard.  Similar  to  the  survey  question  above,  instructors  were  asked  how 
frequently  they  used  other  Army  KM  resources  to  conduct  knowledge  searches.  Respondents 
answering  that  they  used  Army  KM  resources  other  than  BCKS  daily  or  weekly  were  considered 
to  be  using  actively  Army  KM  resources.  Of  the  67  instructors  who  responded  to  this  question 
9%  (N=  6)  indicated  that  they  did  not  search  for  supplemental  information.  Twenty-six  percent 
of  the  remaining  respondents  (16  of  61)  reported  actively  (daily  and  weekly)  using  Army  KM 
resources  for  conducting  knowledge  searches.  Twenty-eight  (17  of  61)  percent  reported  never 
using  Army  KM  resources. 

4.  Percent  of  instructors  who  used  non-Army  resources  to  post  supplemental  course 
content  into  Blackboard.  The  same  survey  question  to  assess  the  above  two  metrics  was  used  to 
assess  this  metric.  Instructors  were  asked  to  indicate  how  important  non-Army  resources,  such  as 
Google  or  Early  Bird,  were  for  them  when  conducting  knowledge  searches  to  enhance  their 
course  content.  Of  the  69  instructors  who  responded  to  this  question,  7%  (5  of  69)  indicated  that 
they  did  not  search  for  supplemental  information.  Seventy-eight  percent  (50  of  64)  reported 
actively  (daily  and  weekly)  using  non-Army  resources  for  conducting  knowledge  searches. 

5.  Perceived  ease  of  updating  course  content  within  the  LVN  LLI  framework.  Instructors 
were  surveyed  to  determine  their  perceptions  of  how  easy  it  was  to  use  Blackboard  to  distribute 
supplements  to  the  formal  ILE  curriculum.  Of  the  60  instructors  who  reported  having  posted  in 
Blackboard,  58%  (N=  35)  indicated  that  that  it  was  fairly  easy  or  easy  to  post  content  and  42% 
reported  some  or  great  difficulty.  Sixty- four  percent  of  the  February- start  instructors  who  posted 
in  Blackboard  (9  of  14)  reported  that  it  was  easy  or  fairly  easy  and  64%  of  the  August-start 
instructors  (23  of  36)  reported  the  same.  Thirty-six  percent  of  both  samples  reported  some  or 
great  difficulty. 

6.  Percent  of  instructors  who  used  Blackboard  to  provide  ‘‘read  ahead"  notes  for  each 
class.  In  a  single  survey  question,  instructors  were  asked  to  identify  which  of  several 
administrative  tasks  they  used  Blackboard  to  conduct.  The  list  of  administrative  tasks  included 
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(a)  making  announcements;  (b)  delivering  handouts;  (c)  coordinating  class  activity  or  group 
projects;  (d)  administering  multiple-choice  exams;  (e)  receiving  course  products  (e.g.,  term 
papers,  presentations,  etc.);  (f)  recording  grades;  and  (g)  none  of  the  above.  Of  the  69  instructors 
who  responded  to  the  survey,  52%  (/V=  36)  reported  using  Blackboard  to  provide  “read  ahead” 
notes  (i.e.,  to  deliver  handouts)  for  each  class.  Forty-nine  percent  (N=  34)  used  Blackboard  to 
make  announcements,  26%  (N=  18)  used  it  to  coordinate  class  activity  or  group  projects,  and 
12%  (N=  8)  used  it  to  receive  course  assignments.  Approximately  one-third  of  the  sample  (N  = 
23)  reported  not  using  Blackboard  for  any  administrative  tasks. 

7.  Percent  of  instructors  who  used  Blackboard  to  administer  course  exams.  As  indicated 
in  instructor  responses  to  the  question  described  above,  10%  (N=  7)  used  Blackboard  to 
administer  course  exams.  One  likely  reason  for  this  low  percentage  is  that  many  of  the  methods 
used  to  assess  student  performance  in  ILE  do  not  involve  multiple-choice  exams. 

8.  Percent  of  instructors  using  Blackboard  to  report  student  grades  to  administrators. 
Ten  percent  of  instructors  (7  of  69)  reported  using  Blackboard  to  report  grades.  Currently, 
Blackboard  is  not  the  official  means  by  which  grades  are  delivered  to  the  college  administration. 
Rather,  a  different  means— the  legacy  data  entry  and  storage  system  used  by  the  administration, 
which  is  not  interoperable  with  Blackboard— is  used.  It  is  likely  for  this  reason  that  instructors 
reported  a  low  adoption  rate  for  the  Blackboard  grade  book.  The  instructors  interviewed  did  not 
see  a  benefit  to  recording  the  same  grades  twice. 

9.  Presence  of  formal  and/or  informal  SOPs  for  leveraging  the  capabilities  of  the 
software  applications  for  course  administration  and  facilitation  (Yes/No).  The  presence  of 
formal  and/or  informal  standard  operating  procedures  was  assessed  by  interviewing  the  technical 
support  staff  and  surveying  and  interviewing  instructors.  The  interviews  indicated  that  SOPs  did 
not  exist  formally  outside  of  the  structure  of  the  Blackboard  site  for  instructors.  Informal  SOP 
development  sometimes  was  accomplished  when  teaching  teams  requested  assistance  from  the 
technical  staff  in  leveraging  Blackboard’s  advanced  capabilities.  When  surveyed  about 
participating  in  the  SOP  development  process,  52%  (34  of  66)  of  instructors  stated  that  there 
were  no  SOPs. 

10.  Percent  of  early  adopters  involved  in  SOP  development.  Instructors  were  surveyed  to 
assess  how  many  were  involved  in  SOP  development.  Early  adopters  were  not  distinguished 
from  late  adopters  in  the  present  research  because  all  instructors  voluntarily  and  actively  using 
Blackboard  in  the  LLC  pilot  and  assisting  in  SOP  development  were  considered  early  adopters. 
Of  the  66  instructors  who  responded  to  this  question,  33%  (N=  22)  reported  being  involved  in 
the  development  process. 

11.  Presence  of  a  formal  and/or  informal  mentorship  system  for  assisting  late  adopters 
leverage  software  capabilities  (Yes/No).  The  presence  of  a  formal  and/or  informal  mentorship 
system  for  instructors  was  assessed  by  interviewing  the  technical  support  staff  and  surveying 
instructors.  Both  sources  of  information  indicated  that  there  was  no  formal  mentorship  program. 
As  described  previously,  informal  mentorship  in  the  form  of  advanced  Blackboard  classes,  was 
provided  by  the  DOET  technical  support  staff  on  an  as-needed  basis.  Individual  instructors  or 
teaching  teams  could  request  advanced  training  to  meet  their  specific  requirements.  Forty  percent 
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(N=  27)  of  the  69  instructors  who  responded  to  the  survey  indicated  that  they  were  occasionally 
or  regularly  involved  in  some  kind  of  informal  group  activity  (e.g.,  discussion  groups)  whose 
intent  was  to  share  knowledge  about  leveraging  Blackboard’s  capabilities. 

Summary.  Overall,  the  instructor  activity  metrics  revealed  that  instructors  have  been  slow 
to  adopt  Blackboard  both  for  posting  course  content  and  for  conducting  administrative  tasks. 
Although  adoption  rates  improved  from  24%  (February-start)  to  34%  (August-start),  a  greater 
level  of  adoption  will  be  necessary  to  optimize  the  impact  Blackboard  could  have  on  teaching 
and  learning.  The  metric  results  do  not  suggest  that  technical  difficulty  is  the  main  barrier  to 
Blackboard  use,  but  rather  that  instructors  have  several  alternatives  readily  available  (and 
perhaps  more  familiar)  to  them  for  conducting  academic  tasks.  For  instance,  instructors  most 
frequently  reported  using  SharePoint  as  their  primary  alternative  to  Blackboard  for  posting 
supplemental  course  content  (39%). 

There  appeared  to  be  interest  among  instructors  to  help  one  another  in  using  Blackboard, 
but  formal  codification  of  what  was  learned  seemed  to  exist  in  pockets.  For  example,  about  as 
many  instructors  stated  that  there  were  no  standard  operating  procedures  for  posting  content  in 
Blackboard  as  there  were  instructors  reporting  that  they  had  participated  in  the  development  of 
just  such  procedures.  Moreover,  the  most  common  mode  of  seeking  help  was  to  ask  a  peer, 
rather  than  to  develop  a  collective  knowledge  base. 

Of  the  knowledge  resources  instructors  rely  on  to  supplement  curriculum  materials,  non- 
Army  resources,  such  as  Google  and  Early  Bird,  by  far  have  the  greatest  frequency  of  access 
(78%  weekly  or  daily  access),  relative  to  Army  KM  resources  (26%  weekly  or  daily  access)  and 
the  BCKS  Leader  Network  in  specific  (4%  weekly  or  daily  access).  This  finding  suggests  a 
disconnect  between  the  operational  Army  and  the  institutional  Army  that  should  be  addressed 
before  the  full  impact  of  the  LVN  LLI  on  course  relevance  can  be  achieved.  However,  the  ability 
to  repair  this  disconnect  is  beyond  the  scope  of  the  LVN  LLI  architects,  and  the  majority  of 
stakeholders  feel  that  course  content  is  already  relevant. 

Selected  Activity  Metrics  -  Leadership 

Six  metrics  capturing  the  activity  of  the  LVN  LLI  leadership  were  measured.  These 
metrics  were  selected  in  order  to  explore  the  efficacy  of  the  leadership  in  overseeing  the 
initiative,  procuring  resources,  and  generating  vision  in  collaboration  with  stakeholders  (see,  e.g., 
DTIC,  2000;  Willis,  2000).  According  to  the  logic  model,  effective  leadership  is  an  important 
determinant  of  stakeholder  adoption  of  instructional  technology,  which  is  critical  for  achieving 
educational  impact.  A  detailed  description  of  each  metric  is  provided  below. 

1.  Stakeholder  perceptions  of  inclusion  in  the  decision-making  process.  This  metric  was 
captured  by  surveying  curriculum  developers  and  instructors,  asking  them  if  they  had 
participated  in  the  decision  making  that  led  to  the  adoption  of  SharePoint  and  Blackboard  as  a 
replacement  for  the  system  they  were  used  to  using.  Seven  out  of  10  (70%)  of  the  curriculum 
developers  indicated  that  they  did  not  provide  input  to  the  decision  making  process.  None  of  the 
three  course  authors  who  did  report  providing  input  were  sure  if  their  input  had  been  used.  Of  the 
67  instructors  who  reported  on  their  involvement  in  the  decision  making,  82%  (55  of  67)  percent 
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indicated  that  they  did  not  provide  input  and  none  of  those  who  reported  providing  input  were 
sure  whether  their  input  had  been  used. 

2.  Percent  of  curriculum  developers  who  understood  the  "commander’s  intent”  for  the 
LLC.  A  single  survey  question  addressed  this  metric.  Curriculum  developers  were  asked  to 
indicate  their  understanding  of  the  LLC  concept  and  were  given  a  series  of  options  from  which  to 
select  the  statement  that  best  reflected  their  understanding.  Seven  of  the  1 0  course  authors  who 
responded  to  this  survey  question  selected  a  definition  that  captured  at  least  some  aspect  of  the 
intent  of  the  LLC  concept.  Three  of  10  selected  a  definition  that  closely  resembled  statements  in 
the  LLC  planning  documentation. 

3.  Percent  of  instructors  who  understood  the  " commander’s  intent"  for  the  LLC.  In  a 
separate  survey,  instructors  were  asked  the  same  question  as  the  curriculum  developers.  Sixty- 
two  percent  (N  =  42)  of  the  67  instructors  who  responded  to  this  survey  question  selected  a 
definition  that  captured  at  least  some  aspect  of  the  intent  of  the  LLC  concept.  Forty-eight  percent 
(N  =  33)  selected  a  definition  that  closely  resembled  statements  made  in  the  LLC  planning 
documentation. 

4.  Adequacy  of funding  for  the  LL1.  This  metric  was  captured  by  an  interview  with  the 
DOET  leadership  in  which  the  leadership  was  asked  about  the  adequacy  of  LLI  funding  and  the 
challenge  areas  in  procuring  funding  for  the  LLI.  The  DOET  leadership  indicated  that  there  was 
no  difficulty  securing  funding  to  meet  the  costs  of  the  LLI  and  that  the  current  funding  levels 
were  adequate  for  meeting  LLI  needs. 

5.  Adequacy  of personnel  for  the  LVN  LLI.  This  metric  was  captured  partially  by  an 
interview  with  the  DOET  leadership  in  which  the  leadership  was  asked  about  the  adequacy  of 
personnel  for  supporting  the  LVN  LLI  and,  if  inadequate,  what  additional  personnel  must  be 
hired  to  effectively  support  the  LVN  LLI.  The  DOET  leadership  indicated  that  current  technical 
staff  levels  were  sufficient  for  meeting  LVN  LLI  needs.  It  also  was  communicated,  however,  that 
down  time  in  the  availability  of  the  LVN  LLI  to  students,  faculty,  and  curriculum  developers  was 
a  result  of  preventable  mistakes  by  technical  personnel.  For  example,  a  week-long  outage  of 
Blackboard  was  attributed  to  the  failure  of  the  technical  staff  to  follow  common  procedures  for 
backing  up  database  contents.  A  45-minute  outage  of  SharePoint  was  attributed  to  the  failure  of 
the  network  administrator  to  maintain  the  appropriate  access  permissions  to  the  Fort 
Leavenworth  domain. 

The  DOET  leadership  emphasized  that  although  the  Web  presence  of  the  LVN  LLI  has 
achieved  very  high  levels  of  stability,  even  minor  downtime  can  have  a  substantial  impact  on 
user  trust  in  the  LVN  LLI’s  ability  to  meet  instructional  needs  and  consequent  use  of  the  system. 
Activities  reported  as  necessary  to  foster  user  trust  after  system  downtime  included  explaining 
the  cause  of  the  downtime  and  the  estimated  time  to  solve  system  problems.  Methods  for  sharing 
this  explanation  included  telephone,  email,  and  face-to-face  meetings. 

Technical  staff  asked  about  the  adequacy  of  the  personnel  supporting  the  LVN  LLI 
indicated  that  there  were  fewer  personnel  available  than  required  for  the  start-up  phase  of  the 
initiative  (8  personnel  at  the  time  of  this  reserarch— as  opposed  to  10— with  3  of  the  eight 
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personnel  hired  on  in  the  summer  of  2006).  Competing  demands  on  limited  staff  time  may 
account  for  the  occasional  errors  leading  to  system  down  time.  However,  the  technical  staff 
indicated  that  the  current  number  of  personnel  would  be  sufficient  to  run  the  LVN  LLI  when  it 
has  achieved  a  steady  state  and  that  staffing  decisions  must  balance  present  and  future  needs. 

6.  Adequacy  of  technology  for  the  LVN  LLI.  This  metric  was  captured  by  an  interview 
with  the  DOET  leadership  in  which  the  leadership  was  asked  about  the  adequacy  of  technology 
for  supporting  the  LVN  LLI  and,  if  inadequate,  what  additional  technology  must  be  procured  to 
effectively  support  the  LVN  LLI.  The  DOET  leadership  indicated  that  the  technology  currently 
available  to  the  DOET  was  adequate  for  meeting  LVN  LLI  needs. 

Summary.  The  LVN  LLI  leadership  has  successfully  procured  resources  to  ensure  the 
adequate  functioning  of  the  technical  aspects  of  the  initiative.  This  emphasis  on  technological 
functioning  is  consistent  with  the  mission  of  the  DOET,  which  was  assigned  to  lead  the 
implementation  of  the  initiative.  There  appear  to  be,  however,  some  less  technical  leadership 
tasks,  arguably  outside  of  the  scope  of  the  DOET,  that  have  not  been  addressed  and  likely  have 
implications  for  system  adoption.  These  interpersonal  leadership  tasks  include  communicating 
the  educational  vision  of  the  LVN  LLI  and  winning  the  stakeholder  buy-in.  A  large  majority 
(81%;  62  of  77)  of  instructors  and  course  authors  reported  that  they  did  not  provide  input  to  the 
decision-making  process  that  has  created  new  work  demands  for  them.  In  addition,  there  is 
evidence  that  nearly  half  (53%,  41  of  77)  of  the  surveyed  stakeholders’  (i.e.,  course  authors  and 
instructors)  were  unaware  of  the  intent  of  TRADOC’s  lifelong  learning  initiative. 

Conclusions  and  Recommendations 


Results  Summary 

Overall,  the  evaluation  results  indicate  that  the  LVN  LLI  is  a  cost-effective  solution  for 
enhancing  the  educational  outreach  of  ILE  curriculum  materials.  For  what  amounts  practically  to 
the  same  amount  of  money,  ILE  delivered  via  the  LVN  LLI  will  eliminate  the  one-year  lag  in 
curriculum  content  between  the  schoolhouse  and  Army  reserve  facilities  and  already  supports 
anytime,  anywhere  learning  for  ILE  students  in  residence  and  at  satellite  campuses.  There  is 
relatively  little  the  LVN  LLI  can  do  to  reduce  the  6-10  month  lag  between  changes  in  the 
operational  environment  and  revisions  to  the  standardized  ILE  curriculum,  but  it  is  assisting  in 
circumventing  this  problem  by  making  it  easier  for  instructors  to  supplement  curriculum 
materials.  The  majority  of  students,  instructors,  and  course  authors  surveyed  reported  that  course 
content  was  relevant  to  the  jobs  of  ILE  graduates. 

The  active  adoption  rate  of  the  LVN  LLI  by  course  authors,  instructors,  and  students  in 
the  schoolhouse  remains  well  below  100%.  Survey  and  interview  data  indicated  that  technical 
difficulty  was  not  the  main  barrier  to  system  use.  Rather,  the  main  barriers  to  system  use  appear 
to  have  been  (a)  ready  availability  of  alternatives  to  the  system  that  were  more  familiar  and 
easier  to  access;  (b)  inconsistent  or  absent  information  management  procedures  necessary  to 
make  posted  content  easy  to  find;  and  (c)  lack  of  instructor  and  course  author  involvement  with 
and  buy-in  to  the  lifelong  learning  concept.  As  has  been  found  in  previous  evaluations  of  this 
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kind  (Cianciolo  et  al.,  2006),  greater  success  has  been  achieved  in  the  technical  implementation 
of  the  initiative  than  in  the  cultivation  of  stakeholder  enthusiasm  and  investment. 

Limited  adoption  of  the  LVN  LLI  did  not  appear  to  hamper  the  academic  experience  of 
ILE  students  and  instructors  in  the  first- year  pilot.  Large  proportions  of  students  reported 
effective  classroom  facilitation  behaviors  by  instructors,  high  levels  of  learning  self-efficacy  and 
motivation,  and  active  involvement  in  their  own  learning  through  independent  study.  The 
majority  of  instructors  indicated  that  the  LVN  LLI  did  not  negatively  impact  classroom 
efficiency,  and  a  small  proportion  of  instructors  reported  that  the  LVN  LLI  enhanced  their 
classroom  efficiency. 

Recommendations 

It  would  be  feasible  to  recommend  that  no  changes  be  made  to  the  LVN  LLI 
implementation.  If  nothing  about  the  LVN  LLI  implementation  were  to  change  from  this  point 
forward,  ILE  instruction  would  continue  successfully,  and  students— especially  those  at  Army 
reserve  facilities— would  benefit  from  greater  access  to  more  relevant  standardized  curriculum, 
all  at  relatively  little  cost  to  the  Army.  Moreover,  adoption  rates  likely  would  increase  on  their 
own  [albeit  slowly  and  probably  not  to  criterion  (i.e.,  100%)],  as  indicated  by  the  generally 
upward  trend  in  adoption  rates  occurring  during  the  first-year  pilot. 

That  said,  the  findings  of  the  present  evaluation  illustrate  the  importance  of  taking  a  logic 
modeling  approach  to  understanding  the  impact  of  lifelong  learning  initiatives.  Assessment  of 
the  LVN  LLI’s  outcomes  alone  would  have  indicated  that  the  initiative  had  achieved  its  goals  but 
would  have  obscured  the  fact  that  a  subset  of  these  goals— teaching  and  learning  effectiveness— 
was  achieved  independently  of  the  use  of  the  LVN  LLI’s  components.  Put  another  way,  the  LVN 
LLI  has  yet  to  have  an  impact  on  enhancing  the  ILE  teaching  and  learning  environment  because 
the  unique  capabilities  of  blended  learning  are  not  yet  being  leveraged.  Comments  made  by 
surveyed  students  who  had  used  Blackboard  as  part  of  their  civilian  education  support  this  point 
with  observations  that  civilian  educators  were  more  effective  at  leveraging  Blackboard’s 
capabilities  to  create  a  true  blended  learning  environment.  In  addition,  survey  responses  indicate 
that  neither  a  majority  of  students  (42%,  104  of  250)  nor  of  instructors  (26%,  16  of  61)  was  yet 
convinced  that  Blackboard  would  enhance  ILE  education.  A  typical  “no  significant  difference” 
between  traditional  and  technology-assisted  learning  may  be  found  in  this  case  (see  Russell, 
1999),  but  this  finding  indicates  that  the  blended  learning  initiative  is  not  living  up  to  its 
capability  to  transform  the  classroom  and  engender  a  cultural  shift  toward  a  lifelong  learning 
orientation. 

In  order  for  the  already  successful  LVN  LLI  to  achieve  optimal  educational  and 
organizational  impact,  stronger  beliefs  in  its  purpose  and  higher  active  adoption  rates  of  its 
components  must  be  achieved.  The  accomplishment  of  this  goal  requires  fostering  the  instructor 
and  course  author  buy-in  necessary  to  learn  and  adopt  unfamiliar  technologies  and  to  role-model 
their  use  for  students.  Improved  buy-in  may  be  developed  through  (a)  engaging  other 
schoolhouse  components  in  viewing  and  cultivating  LVN  LLI-assisted  teaching  as  a  critical 
instructor  competency;  (b)  encouraging  and  shepherding  the  involvement  of  instructors  and 
course  authors  in  the  development  of  information  management  procedures;  and  (c)  spreading 
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strategic  communications  that  explain  the  purpose  and  goals  of  the  system,  that  anticipate 
technical  limitations  of  the  system  (i.e.,  login  requirements),  and  that  immediately  follow  system 
outages. 

Other  Schoolhouse  Components.  As  described  previously,  the  CGSC  DOET  is 
responsible  for  the  implementation,  maintenance,  and  oversight  of  the  LVN  LLI.  However,  a 
visual  inspection  of  the  CGSC  organizational  chart  reveals  that  other  components  of  the 
schoolhouse  must  be  involved  if  a  culture  of  acceptance  and  engagement  in  blended  learning  is 
to  be  infused  throughout  the  college.  Chief  among  these  components  is  the  Faculty  &  Staff 
Development  Office,  which  could  assist  in  developing  instructors’  ability  to  visualize  and 
transform  the  learning  experience  using  Blackboard,  even  involving  Blackboard  in  the  faculty 
development  process  itself.  Similar  assistance  could  be  provided  to  course  authors  using 
SharePoint.  Importantly,  successfully  delivering  blended  learning  must  not  be  viewed  as  a  matter 
of  technology  alone,  but  of  effectively  integrating  educational  need,  instructional  strategy,  and 
advanced  technologies  (Clark,  1994;  DTIC,  2000;  Firdyiwek,  1999;  Willis,  2000). 

Information  Management.  The  vast  majority  of  recommendations  from  surveyed  students 
was  that  the  schoolhouse  choose  either  Blackboard  or  SharePoint  for  posting  content  and 
eliminate  entirely  the  system  not  selected.  In  both  interviews  and  survey  comments,  students 
mentioned  being  confused  by  instructors’  inconsistent  use  of  two  systems  with  overlapping 
capabilities  and  were  unable  to  find  content  as  a  result.  This  confusion  appears  to  have  stemmed 
from  ineffective  or  absent  information  management  procedures  on  the  part  of  faculty  who 
instead  of  using  the  required  system  (Blackboard)  as  the  primary  means  for  posting  content  used 
a  more  familiar  alternative  (SharePoint).  Faculty  and  course  authors  who  communicated 
reluctance  to  use  either  system  via  interview  listed  inadequate  information  management  as  a 
limiting  factor. 

The  DOET  technical  staff  supports  the  development  of  instructor  and  course  author 
facility  with  the  LVN  LLI  technologies,  even  assisting  in  the  development  of  information 
management  strategies  within  teaching  and  authoring  teams  and  modifying  the  LVN  LLI 
interface  to  address  known  information  management  challenges.  However,  the  technical  staff  is 
not  responsible  for  developing  school-wide  information  management  procedures,  and  this 
responsibility  does  not  appear  to  be  held  elsewhere.  Instructors  and  course  authors  informally 
help  each  other,  but  there  is  no  widely-recognized  standard  for  collaboration.  A  formal 
assessment  of  the  information-sharing  strategies  currently  used  by  faculty  and  course  authors 
should  be  conducted  to  understand  user  needs  and  to  reveal  best  practices  that  can  be  broadly 
applied.  Instructors,  curriculum  developers,  and  even  students  should  be  actively  involved  in  the 
process  of  articulating  need  and  should  participate  in  system  interface  design  revisions.  Ideally, 
leadership  of  this  needs  assessment  and  design  process  would  be  executed  by  an  integrated  team 
of  technologists,  faculty  and  staff  developers,  department  chairs,  instructor  and  course  author 
team  leaders,  and  student  representatives  (e.g.,  staff  group  information  officers). 

Strategic  Communications.  Strategic  communications  follow  system  outages,  but  do  not 
appear  to  occur  as  anticipatory  events  that  could  shape  perceptions  and  advance  organizational 
goals.  For  example,  the  CGSC  Circular  350-1,  which  describes  the  CGSC,  its  components,  and 
ILE,  does  not  mention  the  LVN  LLI  or  TRADOC’s  lifelong  learning  initiative  in  general.  A 
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paragraph  or  two  in  the  circular  explaining  the  technologies  and  their  intended  use  would  help 
resident  students  to  understand  what  the  expected  benefit  is  of  the  system,  to  view  themselves  as 
part  of  a  larger  ILE  student  body  that  benefits  from  outreach,  and  perhaps  to  see  that  the  LVN 
LLI  implementation  has  been  made  in  the  best  interests  of  leader  education.  Currently,  a 
minority  of  students  see  the  initiative  as  motivated  by  the  desire  to  improve  education,  but  a  shift 
in  this  perspective  is  necessary  to  empower  the  initiative  to  enhance  organizational  commitment 
and  individual  readiness. 

Strategic  communications  that  anticipate  technical  limitations  also  may  be  helpful  in 
fostering  user  support  and  understanding  when  difficulties  occur.  Such  communications  may 
include  advertising  Blackboard  and  SharePoint  instruction  for  students  and  explaining  the  source 
of  multiple  login  requirements  and  the  reason  for  two  overlapping  systems. 

Concluding  Comments 

Ultimately,  decisions  regarding  the  future  of  an  initiative  are  values  based  (Cukier,  1997). 
This  evaluation  report  cannot  provide  values.  It  is  merely  intended  to  support  the  values-based 
decision  making  of  others  with  objective  data.  The  objective  data  indicate  that  the  LVN  LLI  is 
technically  well  implemented,  cost-effective,  and  enhances  educational  outreach.  The  objective 
data  also  indicate  that  some  investment  is  required  to  strengthen  the  initiative’s  impact  on 
teaching  and  learning.  This  representation  of  the  LVN  LLI  places  it  squarely  within  the 
population  of  technology-assisted  learning  initiatives  working  through  their  first-year  pilot  and 
within  the  population  of  maturing  Army  programs  designed  to  support  human  functioning  with 
technological  means.  The  close  coupling  of  the  LVN  LLI  technologies  with  the  needs  and 
expectations  of  people  who  use  them  will  transform  the  already  successful  initiative  into  a  potent 
educational  and  organizational  multiplier. 
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APPENDIX  B-l 

CURRICULUM  DEVELOPER  SURVEY 


The  curriculum  developer  survey  was  administered  online  for  7  days  in  collaboration 
with  the  CGSC  Quality  Assurance  Office.  Curriculum  developers  who  authored  course  content 
for  either  the  2006  February-start  ILE  class  or  the  2006  August-start  class  (or  both)  received  an 
invitation  via  email  to  participate  in  the  survey.  Ten  curriculum  developers  responded  to  the 
survey,  representing  a  67%  overall  response  rate.  Of  these  10  individuals,  0  had  authored  only 
the  February-start  ILE  curriculum,  1  had  authored  only  the  August-start  ILE  curriculum,  and  9 
had  authored  curriculum  materials  for  both  classes.  Due  to  the  small  number  of  respondents  in 
each  category,  the  data  were  collapsed  across  all  course  authors  for  each  question.  A  complete 
reproduction  of  the  survey  is  shown  below. 

SharePoint  and  Resident  ILE  -  Course  Author  Experience 

The  2006  February-start  and  2006  August-start  CGSOC  classes  are  serving  as  test  cases 
for  using  SharePoint  (instead  of  the  g:  drive)  as  the  primary  means  to  post  and  share  course 
curriculum  materials  during  the  Course  Authoring  process.  Working  within  this  pilot  program, 
you  have  probably  experienced  the  inevitable  challenges  associated  with  transitioning  from  one 
technology  to  another.  Likely,  there  also  have  been  some  unanticipated  benefits  of  the  change. 

The  purpose  of  this  brief,  21 -item  survey  is  to  capture  selected  aspects  of  the  Course 
Author  experience  using  SharePoint  to  post  and  share  common  core  curriculum  materials.  The 
survey  has  3  parts  relating  to: 

1.  Technical  Support; 

2.  Curriculum  Development;  and 

3.  Participation  in  the  Transition  Process. 

The  goal  of  this  survey  is  to  take  the  "pulse"  of  the  transition  effort  —  to  identify  the 
strengths  and  weakness  of  the  technical  support  provided  to  Course  Authors  as  well  as  the  actual 
level  of  Course  Author  use  of  SharePoint.  The  survey  also  is  intended  to  reveal  the  level  of 
participation  Course  Authors  perceive  they  have  had  in  the  process  of  transitioning  from  the  g: 
drive  to  SharePoint. 

This  survey  is  completely  anonymous  and  voluntary.  Your  responses  are  in  no  way 
associated  with  your  name  and  you  may  choose,  without  consequence,  not  to  participate. 

As  you  fill  out  this  survey,  consider  your  experiences  in  developing  curriculum  materials 
for  the  2006  February-start  or  the  2006  August-start  ILE  CC.  Select  the  response  that  best 
describes  your  experiences  associated  with  each  statement.  Also,  there  are  optional  open  ended 
questions  at  the  end  of  each  section  to  add  comments. 

If  you  have  any  questions  concerning  the  content  of  this  survey,  contact  Dr.  Anna 
Cianciolo,  217-621-3918.  If  you  experience  technical  difficulties  with  it,  contact  Mr.  Douglas 
Loa,  913-684-7275. 
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This  Survey  has  been  approved  by  QAO  and  the  Survey  Control  number  is  06-080 

Section  I  -  Technical  Support 

1 .  How  accessible  did  you  find  the  technical  support  to  be? 

( )  What  tech  support?  I  didn't  know  there  was  tech  support. 

( )  I  knew  how  to  find  the  tech  support  team,  but  actually  getting  a  hold  of  someone  was 
unpredictable;  on  more  than  one  occasion  I  had  to  wait  longer  than  I  would  have  liked. 

( )  I  wouldn't  know;  I  knew  there  was  a  tech  support  team,  but  1  never  tried  to  contact 
anybody. 

( )  I  generally  didn't  have  trouble  finding  or  connecting  with  someone  in  tech  support;  if  I 
ever  had  to  wait,  it  wasn't  a  big  deal. 

(  )  Finding  and  connecting  with  tech  support  was  no  problem  at  all;  I  always  got 
somebody  when  I  needed  them. 

2.  How  useful  did  you  find  the  technical  support  to  be? 

(  )  Not  at  all  useful;  I  had  to  find  other  ways  to  fix  my  problems." 

( )  They  solved  technical  problems  ok,  but  if  the  tech  support  was  better,  the  kinds  of 
problems  1  had  wouldn't  exist. 

( )  I  wouldn't  know;  I  never  asked  for  tech  support. 

( )  I  had  a  pretty  good  experience;  the  system  worked  well  enough  and  when  problems 
did  come  up,  they  were  usually  fixed  the  first  time  around. 

( )  They're  the  best;  the  system  worked  great  and  whenever  I  needed  technical  assistance, 
it  was  provided  quickly,  effectively,  and  professionally. 

3.  How  accessible  did  you  find  the  SharePoint  help  documentation  to  be? 

(  )  What  help  documentation?  I  didn't  know  there  was  help  documentation. 

( )  I  think  there  was  supposed  to  be  help  documentation,  but  I  sure  couldn't  find  it. 

( )  I  wouldn't  know;  I  never  looked  for  help  documentation. 

( )  I  could  usually  find  help  documentation  on  the  topics  1  needed  to  know  about. 

(  )  I  could  always  find  the  topic  I  was  looking  for  in  the  help  documentation. 

4.  How  useful  did  you  find  the  SharePoint  help  documentation  to  be? 

()  Not  at  all  useful;  I  don't  even  bother  looking  anymore. 

( )  It  was  ok  for  the  topics  it  handled,  but  it  usually  didn't  handle  the  topics  I  needed  help 
in. 

( )  I  wouldn't  know;  I  never  used  the  help  documentation. 

( )  It  generally  provided  useful  information,  but  I  don't  think  it  handled  everything. 

( )  It  always  met  my  information  needs. 


112 


5.  How  usable  (readable)  did  you  find  the  SharePoint  help  documentation  to  be? 

( )  The  help  documentation  was  mostly  gibberish  written  by  someone  who  knows  the 
system  but  not  how  to  write. 

()  It  was  ok;  I  usually  had  to  read  it  a  couple  of  times  over  to  get  what  it  was  saying,  but 
eventually  I  understood  it. 

( )  I  wouldn't  know;  I  never  used  the  help  documentation. 

( )  I  usually  didn't  have  trouble  understanding  the  help  documentation,  but  I'm  used  to 
reading  technical  writing. 

( )  The  help  documentation  was  easy  to  understand;  even  my  grandmother  would  get  it! 

6.  Have  you  participated  in  SharePoint  instruction? 

()  Yes 

( )  No  - 1  already  know  how  to  use  SharePoint 
( )  No  - 1  didn't  know  there  was  SharePoint  instruction 
( )  No  - 1  didn't  have  time 

6a.  Did  you  have  to  re-leam  anything  you  were  taught  in  order  to  perform  instructional  tasks? 

()  Yes 
( )  No 

Additional  Comments  regarding  the  Technical  Support  (optional): 

Section  II  -  Curriculum  Development 

7.  How  knowledgeable  do  you  feel _ are  with  using  SharePoint? 

your  students 

( )  Very  knowledgeable 
( )  Knowledgeable 
( )  Somewhat  knowledgeable 
( )  Very  little 
( )  Not  at  all 


you 


( )  Very  knowledgeable 
( )  Knowledgeable 
( )  Somewhat  knowledgeable 
( )  Very  little 
( )  Not  at  all 
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8.  When  you  distributed  curriculum  materials  to  other  Course  Authors,  what  means  did  you  use 
most  often? 

( )  SharePoint 
( )  g:  drive 

( )  Distributed  via  email 
( )  Distributed  via  memory  stick/thumb  drive 
( )  Printed  materials 

( )  Other  (e.g.,  AKO  personal  webpage)  {  ] 

9.  How  easy  was  it  to  post  and  locate  curriculum  materials  in  SharePoint? 

( )  It  was  very  difficult;  I  mostly  used  other  means  to  share  resources  with  Course 
Authors. 

( )  Let's  just  say  it  wasn't  the  g  drive;  it  limited  my  options  for  sharing  information  with 
the  people  who  I  felt  needed  it. 

( )  No  idea;  I  never  posted  in  SharePoint. 

( )  Fairly  easy,  but  the  g:  drive  is  more  flexible. 

( )  Piece  of  cake;  I  had  no  problems  at  all. 

For  the  following  question,  consider  your  experience  in  past  years  writing  ILE  common  core 
curriculum  materials. 

1 0.  How  frequently  were  you  able  to  update/revise  course  content  based  on  new  developments  in 
the  field,  instructor  or  student  feedback,  etc.? 

(  )  Regularly;  prior  to  each  course  administration. 

( )  Semi-regularly;  prior  to  most  administrations  of  the  course,  but  not  all. 

( )  Infrequently;  not  often  enough  to  do  much  good. 

( )  Never;  revisions  are  so  out  of  synch  with  course  administrations  that  I  don't  even  try 
anymore. 

( )  N/A;  I've  only  worked  on  the  core  curriculum  once. 

For  the  following  question,  consider  your  experience  writing  ILE  common  core  curriculum 
materials  since  the  introduction  of  SharePoint. 

1 1 .  How  has  using  SharePoint  changed  the  frequency  with  which  you  update/revise  course 
materials? 

( )  Frequency  greatly  decreased;  Time  that  was  spent  modifying  course  content  is  now 
spent  trying  to  find  and/or  store  files  on  SharePoint. 

( )  Frequency  somewhat  decreased;  SharePoint  makes  it  harder  for  Course  Authors  to 
share  the  resources  that  would  enhance  or  synchronize  course  content. 

( )  Not  at  all;  Delays  in  revising  course  content  have  nothing  to  do  with  technology  -  they 
are  administrative  in  nature. 

( )  Frequency  somewhat  increased;  SharePoint  makes  it  easier  for  Course  Authors  to 
share  the  resources  that  would  enhance  or  synchronize  course  content. 
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( )  Frequency  greatly  increased;  One-stop  access  to  professional  forums,  Army 
knowledge  repositories,  and  other  Course  Authors  has  revolutionized  the  way  we  do 
business. 

( )  N/A;  I've  only  worked  on  the  core  curriculum  once. 

Additional  Comments  regarding  Curriculum  Development.(optional): 

12.  The  SharePoint  Learning  System  enhances  your  students  ILE  education. 

( )  Strongly  Agree 
( )  Agree 
( )  Undecided 
( )  Disagree 
( )  Strongly  Disagree 

13.  How  easy  was  it  to  access  SharePoint  from  outside  the  schoolhouse? 

( )  I  tried  to  work  from  home  (or  away),  but  never  could  get  access. 

( )  I  could  occasionally  get  access  to  SharePoint,  but  not  often  enough  to  make  working 
from  outside  the  schoolhouse  a  viable  option. 

( )  I  wouldn't  know;  I  never  tried  to  access  SharePoint  from  outside  the  schoolhouse. 

()  I  was  able  to  get  access  to  SharePoint  from  outside  the  schoolhouse  when  I  needed  it, 
but  I  don't  see  being  able  to  work  from  outside  the  schoolhouse  as  a  big  advantage. 

( )  Very  easy;  I  had  no  problem  with  access  and  it  enabled  me  to  get  valuable  work  done 
when  I  otherwise  would  not  have  been  able  to  do  it. 

14.  How  current  or  relevant  did  you  feel  the  core  curriculum  you  developed  was  to  the  upcoming 
job  demands  of  the  ILE  students? 

( )  Not  at  all  relevant 

()  It  was  no  less  relevant  than  one  can  expect  given  the  rapidly  changing  operational 
environment  and  restrictions  on  using  FOUO  content  in  the  classroom 
( )  It  covered  relevant  topics,  but  remember  our  focus  is  on  education,  not  mission 
preparation 

()  It  was  definitely  more  relevant  than  it  used  to  be,  but  we're  not  there  yet 
( )  Totally  relevant;  Graduates  will  be  able  to  arrive  in  their  units  completely  prepared  to 
fulfill  the  requirements  of  their  duty  position 

15.  When  you  searched  for  current  information  to  write  curriculum  materials,  how  would  you 
rate  the  importance  of  the  following  resources: 

Professional  forums  in  the  Battle  Command  Knowledge  System  (BCKS)  Leader  Network 

( )  Extremely  Important 
( )  Somewhat  important 
( )  Equal  Importance 
( )  Not  too  important 
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( )  Not  important  at  all 
( )  Didn't  search  for  current  information 


Army  knowledge  repositories  (e.g.,  AKO,  CALL,  etc.) 

( )  Extremely  Important 

( )  Somewhat  important 

( )  Equal  Importance 

()  Not  too  important 

( )  Not  important  at  all 

( )  Didn't  search  for  current  information 

Non-Army  resources  (e.g.,  Google,  Early  Bird,  etc.) 

( )  Extremely  Important 
( )  Somewhat  important 
( )  Equal  Importance 
( )  Not  too  important 
( )  Not  important  at  all 
( )  Didn't  search  for  current  information 

Section  III  -  Participation  in  the  Transition  Process 

16.  Did  you  assist  in  developing  standard  operating  procedures  within  your  department  and/or 
block  for  posting  and  organizing  content  in  SharePoint? 

( )  What  standard  operating  procedures?  There  are  none  to  my  knowledge. 

( )  I  was  asked  to  help  with  developing  standard  operating  procedures  but  didn't  have 
time  to  get  involved;  I'm  not  sure  if  they  were  ever  completed. 

( )  I  believe  we  have  standard  operating  procedures,  but  they  came  from  the  tech  support 
people,  not  the  instructors. 

( )  I  was  involved  in  developing  the  standard  operating  procedures,  but  they  don't  cover 
everything  and  aren't  always  followed. 

( )  I  participated  fully  in  the  development  of  standard  operating  procedures;  they  are 
widely  available  and  followed. 

17.  Do  you  feel  that  there  are  enough  faculty  to  teach  ILE,  given  the  additional  classes  enabled 
by  delivering  courseware  on  the  web? 

()  Yes 
0  No 

18.  Did  you  participate  in  formal  or  informal  mentoring  among  Course  Authors  on  using 
SharePoint  to  share  curriculum  materials  (either  as  a  mentor  or  a  "mentee")? 

( )  There  was  no  mentoring— formal  or  informal— among  Course  Authors. 
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( )  I  believe  Course  Authors  did  help  each  other  with  SharePoint,  but  I  was  never 
involved. 

( )  Sometimes  if  I  needed  help,  I  would  ask  another  Course  Author/Sometimes  if 
someone  needed  help,  they  would  ask  me. 

( )  Every  once  in  a  while,  a  few  of  us  would  have  a  conversation  about  how  using 
SharePoint  could  change  the  way  we  teach. 

( )  I  was  regularly  involved  in  helping  others  leverage  SharePoint's  capabilities. 

1 9.  Which  of  the  following  classes  did  you  author  for? 

February  Start 

()  Yes 

( )  No 

August  Start 

( )  Yes 

( )  No 

20.  SharePoint  is  one  component  of  the  Fort  Leavenworth  Lifelong  Learning  Center  (LLC). 

What  is  your  understanding  of  what  this  LLC  is? 

(  )  Never  heard  of  it. 

( )  It's  a  new  learning  initiative  pushed  down  by  higher  headquarters. 

( )  It's  a  way  to  conduct  resident  and  satellite  instruction  using  the  same  course  materials 
so  all  majors  can  get  the  same  core  intermediate-level  education. 

( )  It's  an  online  repository  of  curriculum  materials  meant  to  support  anytime/anywhere 
access  to  the  institution. 

( )  It's  a  networked,  collaborative  environment  designed  to  link  the  operating  and 
generating  forces  through  common  access  to  course  materials,  emerging  doctrine,  and 
discussion  forums. 

21 .  Did  you  participate  in  the  decision-making  process  that  led  to  the  requirement  to  use 

SharePoint  instead  of  the  g  drive? 

( )  I  had  no  idea  they  were  considering  a  change;  soliciting  Course  Author  input  would 
have  been  helpful. 

( )  I  wasn't  included  in  the  decision  making,  but  I  don't  feel  I  needed  to  be. 

( )  I  was  made  aware  that  a  change  was  being  considered  as  an  alternative  to  the  g  drive, 
but  I  didn’t  provide  any  input  to  the  decision-making  process. 

( )  Somebody  asked  me  what  I  felt  about  using  something  other  than  the  g:  drive,  but  I'm 
not  sure  my  input  really  made  a  difference  one  way  or  the  other. 

( )  I  had  full  visibility  on  the  decision-making  process  and  felt  my  input  was  received  and 
considered. 

22.  What  would  you  like  the  CGSC  leadership  to  know  about  any  aspect  of  the  SharePoint 

System? 
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This  completes  the  survey. 


Again,  thank  you  for  participating  in  this  survey.  Your  time  and  responses  are  valued. 
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APPENDIX  B-2 
INSTRUCTOR  SURVEY 


The  instructor  survey  was  administered  online  for  9  days  in  collaboration  with  the  CGSC 
Quality  Assurance  Office.  Instructors  who  taught  either  the  2006  February-start  ILE  class  or  the 
2006  August-start  class  (or  both)  received  an  invitation  via  email  to  participate  in  the  survey. 
Sixty-nine  of  201  instructors  responded  to  the  survey,  representing  a  34%  overall  response  rate. 
Of  these  69  individuals,  25%  (N  -  17)  had  taught  only  for  the  February-start  ILE  class,  59%  (N  = 
41)  had  taught  only  for  the  August-start  ILE  class,  and  13%  had  taught  both  classes  (N=  6).  Two 
respondents  (3%)  reported  having  taught  neither  class,  but  had  used  Blackboard  for  teaching 
other  components  of  the  CGSOC. 

Blackboard,  SharePoint  and  Resident  ILE  -  Instructor  Experience 

The  2006  February-start  and  2006  August-start  CGSOC  classes  are  serving  as  test  cases 
for  using  Blackboard  (instead  of  the  g:  drive)  as  the  primary  means  of  distributing  curriculum 
materials.  Working  within  this  pilot  program,  you  have  probably  experienced  the  inevitable 
challenges  associated  with  transitioning  from  one  technology  to  another.  Likely,  there  also  have 
been  some  unanticipated  benefits  of  the  change. 

The  purpose  of  this  brief,  23-item  survey  is  to  capture  selected  aspects  of  the  instructor 
experience  using  Blackboard  to  deliver  the  ILE  CC  curriculum  and  to  conduct  classroom 
education.  The  survey  has  4  parts  relating  to: 

1.  Technical  Support; 

2.  Online  Course  Administration; 

3.  Classroom  Course  Administration;  and 

4.  Participation  in  the  Transition  Process. 

The  goal  of  this  survey  is  to  take  the  "pulse"  of  the  transition  effort  —  to  identify  the 
strengths  and  weakness  of  the  technical  support  provided  to  instructors  as  well  as  the  actual  level 
of  instructor  use  of  Blackboard.  The  survey  also  is  intended  to  reveal  the  level  of  participation 
instructors  perceive  they  have  had  in  the  process  of  transitioning  from  the  g:  drive  to  Blackboard. 

This  survey  is  completely  anonymous  and  voluntary.  Your  responses  are  in  no  way 
associated  with  your  name  and  you  may  choose,  without  consequence,  not  to  participate. 

As  you  fill  out  this  survey,  consider  your  experiences  in  delivering  instruction  for  the 
2006  February-start  or  the  2006  August-start  ILE  CC.  Select  the  response  that  best  describes 
your  experiences  associated  with  each  statement.  Also,  there  are  optional  open  ended  questions 
at  the  end  of  each  section  to  add  comments. 

If  you  have  any  questions  concerning  the  content  of  this  survey,  contact  Dr.  Anna 
Cianciolo,  217-621-3918.  If  you  experience  technical  difficulties  with  it,  contact  Mr.  Douglas 
Loa,  913-684-7275. 
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This  Survey  has  been  approved  by  QAO  and  the  Survey  Control  number  is  06-078 

Section  I  -  Technical  Support 

1 .  How  accessible  did  you  find  the  technical  support  to  be? 

( )  What  tech  support?  I  didn't  know  there  was  tech  support. 

( )  I  knew  how  to  find  the  tech  support  team,  but  actually  getting  a  hold  of  someone  was 
unpredictable;  on  more  than  one  occasion  I  had  to  wait  longer  than  I  would  have  liked. 

( )  I  wouldn't  know;  I  knew  there  was  a  tech  support  team,  but  I  never  tried  to  contact 
anybody. 

( )  I  generally  didn't  have  trouble  finding  or  connecting  with  someone  in  tech  support;  if  I 
ever  had  to  wait,  it  wasn't  a  big  deal. 

( )  Finding  and  connecting  with  tech  support  was  no  problem  at  all;  I  always  got 
somebody  when  I  needed  them. 

2.  How  useful  did  you  find  the  technical  support  to  be? 

( )  Not  at  all  useful;  I  had  to  find  other  ways  to  fix  my  problems." 

( )  They  solved  technical  problems  ok,  but  if  the  tech  support  was  better,  the  kinds  of 
problems  I  had  wouldn't  exist. 

(  )  I  wouldn't  know;  I  never  asked  for  tech  support. 

(  )  I  had  a  pretty  good  experience;  the  system  worked  well  enough  and  when  problems 
did  come  up,  they  were  usually  fixed  the  first  time  around. 

( )  They're  the  best;  the  system  worked  great  and  whenever  I  needed  technical  assistance, 
it  was  provided  quickly,  effectively,  and  professionally. 

3.  How  accessible  did  you  find  the  Blackboard  help  documentation  to  be? 

( )  What  help  documentation?  I  didn't  know  there  was  help  documentation. 

( )  I  think  there  was  supposed  to  be  help  documentation,  but  I  sure  couldn't  find  it. 

()  I  wouldn't  know;  I  never  looked  for  help  documentation. 

( )  I  could  usually  find  help  documentation  on  the  topics  I  needed  to  know  about. 

( )  I  could  always  find  the  topic  I  was  looking  for  in  the  help  documentation. 

4.  How  useful  did  you  find  the  Blackboard  help  documentation  to  be? 

( )  Not  at  all  useful;  I  don't  even  bother  looking  anymore. 

()  It  was  ok  for  the  topics  it  handled,  but  it  usually  didn't  handle  the  topics  I  needed  help 
in. 

( )  I  wouldn't  know;  I  never  used  the  help  documentation. 

()  It  generally  provided  useful  information,  but  I  don't  think  it  handled  everything. 

()  It  always  met  my  information  needs. 
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5.  How  usable  (readable)  did  you  find  the  Blackboard  help  documentation  to  be? 

( )  The  help  documentation  was  mostly  gibberish  written  by  someone  who  knows  the 
system  but  not  how  to  write. 

( )  It  was  ok;  I  usually  had  to  read  it  a  couple  of  times  over  to  get  what  it  was  saying,  but 
eventually  I  understood  it. 

( )  I  wouldn't  know;  I  never  used  the  help  documentation. 

( )  I  usually  didn't  have  trouble  understanding  the  help  documentation,  but  I'm  used  to 
reading  technical  writing. 

( )  The  help  documentation  was  easy  to  understand;  even  my  grandmother  would  get  it! 

6.  Have  you  participated  in  Blackboard  instruction? 

()  Yes 

( )  No  - 1  already  know  how  to  use  Blackboard 
( )  No  - 1  didn't  know  there  was  Blackboard  instruction 
( )  No  - 1  didn't  have  time 

6a.  Did  you  have  to  re-leam  anything  you  were  taught  in  order  to  perform  instructional  tasks? 

()  Yes 
( )  No 

Additional  Comments  regarding  the  Technical  Support  (optional): 

Section  II  -  Online  Course  Administration 

7.  How  knowledgeable  do  you  feel _ _ are  with  using  Blackboard? 

your  students 

( )  Very  knowledgeable 
( )  Knowledgeable 
( )  Somewhat  knowledgeable 
( )  Very  little 
( )  Not  at  all 


you 


( )  Very  knowledgeable 
( )  Knowledgeable 
( )  Somewhat  knowledgeable 
( )  Very  little 
( )  Not  at  all 
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8.  When  you  distributed  supplemental  curriculum  materials,  what  means  did  you  use  most  often? 

( )  Blackboard 
( )  SharePoint 
( )  g:  drive 

( )  Distributed  via  email 
( )  Distributed  via  memory  stick/thumb  drive 
( )  Printed  materials 

( )  Other  (e.g.,  AKO  personal  webpage)  [  ] 

9.  How  easy  was  it  to  post  supplemental  curriculum  materials  in  Blackboard? 

()  It  was  very  difficult;  I  mostly  used  other  means  to  share  resources  with  my  students. 

( )  Let's  just  say  it  wasn't  the  g  drive;  it  limited  my  options  for  sharing  information  with 
the  people  who  I  felt  needed  it. 

()  No  idea;  I  never  posted  in  Blackboard. 

( )  Fairly  easy,  but  the  g:  drive  is  more  flexible. 

( )  Piece  of  cake;  I  had  no  problems  at  all. 

10.  How  frequently  did  you  supplement  the  core  curriculum  materials  based  on  current  events 
(for  example,  by  distributing  a  relevant  news  story  or  emerging  doctrine)? 

( )  Daily 
( )  Weekly 

( )  Every  couple  of  weeks 
( )  Monthly 
( )  Never 

1 1 .  How  frequently  did  you  supplement  the  core  curriculum  materials  relative  to  past  ILE 
courses  when  you  used  the  g:  drive  for  posting  course  content? 

(  )  A  lot  more  frequently 
(  )  Somewhat  more  frequently 
(  )  Pretty  much  the  same 
( )  Somewhat  less  frequently 
()  A  lot  less  frequently 

( )  N/A  - 1  don't  supplement  curriculum  materials 
( )  N/A  -  This  is  my  first  time  teaching  the  course 

12.  Which  of  the  following  administrative  tasks  did  you  use  Blackboard  to  conduct  (check  all 
that  apply)? 

( )  Making  announcements  (e.g.,  for  changes  to  readings) 

( )  Delivering  handouts  (e.g.,  supplemental  read-aheads) 

(  )  Coordinating  class  activity  or  group  projects 
(  )  Administering  (multiple-choice)  exams 

(  )  Receiving  course  products  (e.g.,  term  papers,  presentations,  etc.) 


122 


( )  Recording  grades 
( )  None  of  the  above 

13.  When  you  searched  for  current  information  to  supplement  the  curriculum  materials,  how 
often  did  you  use: 

Professional  forums  in  the  Battle  Command  Knowledge  System  (BCKS)  Leader  Network 

( )  Daily 
( )  Weekly 

( )  Every  couple  of  weeks 
( )  Monthly 
( )  Never 

( )  Didn’t  search  for  supplemental  information 
Army  knowledge  repositories  (e.g.,  AKO,  CALL,  etc.) 

( )  Daily 
( )  Weekly 

( )  Every  couple  of  weeks 
( )  Monthly 
( )  Never 

(  )  Didn’t  search  for  supplemental  information 
Non-Army  resources  (e.g.,  Google,  Early  Bird,  etc.) 

( )  Daily 
( )  Weekly 

( )  Every  couple  of  weeks 
( )  Monthly 
( )  Never 

( )  Didn’t  search  for  supplemental  information 
Additional  Comments  regarding  the  Online  Course  Administration  (optional) 

Section  III  -  Course  Administration 

14.  The  Blackboard  Learning  System  enhances  your  students  ILE  education. 

( )  Strongly  Agree 
( )  Agree 
( )  Undecided 
( )  Disagree 
( )  Strongly  Disagree 
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15.  How  much  would  you  estimate  that  doing  administrative  tasks  in  Blackboard  reduced  the 
time  you  had  to  spend  on  administration  in  class? 

( )  I  actually  had  to  spend  more  class  time  on  admin  because  students  often  had  problems 
with  the  system. 

( )  No  real  reduction  in  in-class  admin  time. 

( )  Some  reduction  in  in-class  admin  time;  probably  there  will  be  a  greater  reduction 
when  I  learn  more  about  how  to  use  the  system. 

( )  Reduced  the  time  to  zero;  I  did  not  have  to  do  admin  tasks  in  class,  so  we  devoted 
class  time  entirely  to  learning. 

16.  How  current  or  relevant  did  you  feel  the  core  curriculum  (with  supplements)  was  to  the 
upcoming  job  demands  of  your  students? 

( )  Not  at  all  relevant 

( )  It  was  no  less  relevant  than  one  can  expect  given  the  rapidly  changing  operational 
environment  and  restrictions  on  using  FOUO  content  in  the  classroom 

( )  It  covered  relevant  topics,  but  remember  our  focus  is  on  education,  not  mission 
preparation 

( )  It  was  definitely  more  relevant  than  it  used  to  be,  but  we're  not  there  yet 

( )  Totally  relevant;  Graduates  will  be  able  to  arrive  in  their  units  completely  prepared  to 
fulfill  the  requirements  of  their  duty  position 

Additional  Comments  regarding  the  Course  Administration  (optional): 

Section  IV  -  Participation  in  the  Transition  Process 

17.  Did  you  assist  in  developing  standard  operating  procedures  within  your  teaching 
team/section  for  posting,  organizing,  and  sharing  (e.g.,  across  staff  groups)  content  in 
Blackboard? 

( )  What  standard  operating  procedures?  There  are  none  to  my  knowledge. 

(  )  I  was  asked  to  help  with  developing  standard  operating  procedures  but  didn't  have 
time  to  get  involved;  I'm  not  sure  if  they  were  ever  completed. 

(  )  I  believe  we  have  standard  operating  procedures,  but  they  came  from  the  tech  support 
people,  not  the  instructors. 

( )  I  was  involved  in  developing  the  standard  operating  procedures,  but  they  don't  cover 
everything  and  aren't  always  followed. 

( )  I  participated  fully  in  the  development  of  standard  operating  procedures;  they  are 
widely  available  and  followed. 

18.  Do  you  feel  that  there  are  enough  faculty  to  teach  ILE,  given  the  additional  classes  enabled 
by  delivering  courseware  on  the  web? 

()  Yes 

( )  No 
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19.  Did  you  participate  in  formal  or  informal  mentoring  among  instructors  on  using  Blackboard 

for  instruction  (either  as  a  mentor  or  a  "mentee")? 

( )  There  was  no  mentoring— formal  or  informal— among  instructors. 

()  I  believe  instructors  did  help  each  other  with  Blackboard,  but  I  was  never  involved. 

( )  Sometimes  if  I  needed  help,  I  would  ask  another  instructor/Sometimes  if  someone 
needed  help,  they  would  ask  me. 

( )  Every  once  in  a  while,  a  few  of  us  would  have  a  conversation  about  how  using 
Blackboard  could  change  the  way  we  teach. 

( )  I  was  regularly  involved  in  helping  others  leverage  Blackboard's  capabilities. 

20.  Which  of  the  following  classes  did  you  teach  in? 

February  Start 

()  Yes 

( )  No 

August  Start 

()  Yes 
( )  No 

21 .  Blackboard  is  one  component  of  the  Fort  Leavenworth  Lifelong  Learning  Center  (LLC). 

What  is  your  understanding  of  what  this  LLC  is? 

( )  Never  heard  of  it. 

( )  It's  a  new  learning  initiative  pushed  down  by  higher  headquarters. 

(  )  It's  a  way  to  conduct  resident  and  satellite  instruction  using  the  same  course  materials 
so  all  majors  can  get  the  same  core  intermediate-level  education. 

( )  It's  an  online  repository  of  curriculum  materials  meant  to  support  anytime/anywhere 
access  to  the  institution. 

( )  It's  a  networked,  collaborative  environment  designed  to  link  the  operating  and 
generating  forces  through  common  access  to  course  materials,  emerging  doctrine,  and 
discussion  forums. 

22.  Did  you  participate  in  the  decision-making  process  that  led  to  the  requirement  to  use 

Blackboard  instead  of  the  g  drive? 

( )  I  had  no  idea  they  were  considering  a  change;  soliciting  instructor  input  would  have 
been  helpful. 

( )  I  wasn't  included  in  the  decision  making,  but  I  don't  feel  I  needed  to  be. 

( )  I  was  made  aware  that  a  change  was  being  considered  as  an  alternative  to  the  g  drive, 
but  I  didn't  provide  any  input  to  the  decision-making  process. 

( )  Somebody  asked  me  what  I  felt  about  using  something  other  than  the  g:  drive,  but  I'm 
not  sure  my  input  really  made  a  difference  one  way  or  the  other. 

( )  I  had  full  visibility  on  the  decision-making  process  and  felt  my  input  was  received  and 
considered. 
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23.  What  would  you  like  the  CGSC  leadership  to  know  about  any  aspect  of  the  Blackboard 
System? 


This  completes  the  survey. 


Again,  thank  you  for  participating  in  this  survey.  Your  time  and  responses  are  valued. 
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APPENDIX  B-3 
STUDENT  SURVEY 


The  student  survey  was  administered  online  for  approximately  three  weeks  in 
collaboration  with  the  CGSC  Quality  Assurance  Office.  Students  in  the  2006  February-start  ILE 
class  and  the  2006  August-start  class  received  an  invitation  via  email  to  participate  in  the  survey. 
Two-hundred  fifty-one  students  responded  to  the  survey,  representing  a  30%  overall  response 
rate.  A  complete  reproduction  of  the  survey  is  shown  below. 

Blackboard,  SharePoint  and  Resident  ILE  -  Student  Experience 

Your  resident  CGSOC  ILE  class  is  among  the  first  to  use  Blackboard  as  the  primary 
means  of  distributing  curriculum  materials  and  SharePoint  as  the  primary  means  of  course 
administration.  As  students  during  this  pilot,  you  have  probably  experienced  the  inevitable 
challenges  associated  with  transitioning  from  one  technology  to  another.  Likely,  there  also  have 
been  some  unanticipated  benefits  of  the  change.  The  purpose  of  this  brief,  22-item  survey  is  to 
capture  selected  aspects  of  the  ILE  CC  student  experience  relating  to: 

1 .  Course  content; 

2.  Course  Administration;  and 

3.  The  Overall  Learning  Experience. 

The  goal  of  this  survey  is  to  take  the  "pulse"  of  the  pilot  effort  —  to  identify  how 
Blackboard  and  SharePoint  are  used  by  students  and  to  understand  the  impact  of  these 
technologies  on  course  effectiveness  and  learning. 

This  survey  is  completely  anonymous  and  voluntary.  Your  responses  are  in  no  way 
associated  with  your  name  and  you  may  choose,  without  consequence,  not  to  participate. 

As  you  fill  out  this  survey,  consider  your  experiences  during  ILE  CC.  Select  the  response 
that  best  describes  your  experiences  associated  with  each  statement.  There  is  an  open  ended 
comment  box  at  the  end  of  the  each  section;  please  feel  free  to  add  comments. 

If  you  have  any  questions  concerning  the  content  of  this  survey,  contact  Dr.  Anna 
Cianciolo,  217-621-3918.  If  you  experience  technical  difficulties  with  it,  contact  Mr.  Douglas 
Loa,  913-684-7275. 

This  Survey  has  been  approved  by  QAO  and  the  Survey  Control  number  is  06-077 
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Blackboard,  SharePoint,  and  Resident  Intermediate-Level  Education  (ILE): 

A  Survey  of  the  Student  Experience 

Section  I  -  Course  Content 

1 .  How  relevant  was  the  common  core  curriculum  to  the  job  you  expect  to  have  after  you 
graduate? 

(  )  Not  at  all  relevant. 

(  )  More  irrelevant  than  relevant;  some  topics  addressed  my  needs  and  interests,  but  I 
won't  be  able  to  apply  directly  what  I  learned. 

( )  No  less  relevant  than  any  other  academic  experience;  school  learning  is  supposed  to 
be  "just  in  case"  not  "just  in  time." 

( )  More  relevant  than  irrelevant;  topics  generally  addressed  my  needs  and  interests,  and 
with  some  effort  I  should  be  able  to  apply  what  I  learned. 

( )  Totally  relevant;  I  will  be  fully  prepared  for  my  next  job. 

2.  How  relevant  were  the  supplemental  materials  (e.g.,  readings,  video  interviews,  etc.)  your 
instructors  provided  to  the  job  you  expect  to  have  after  you  graduate? 

(  )  Not  at  all  relevant;  they  just  didn't  cover  topics  of  interest  to  me. 

( )  More  irrelevant  than  relevant;  they  covered  relevant  topics  but  at  too  abstract  a  level 
to  be  useful. 

()  No  less  relevant  than  one  can  expect  in  the  academic  environment  and  with 
restrictions  on  FOUO  content  sharing. 

( )  More  relevant  than  irrelevant;  there  were  some  really  useful  resources,  but  I  could 
have  used  more  of  them. 

( )  Totally  relevant;  I  know  I'll  want  to  reach  back  from  the  field  to  ILE  courses  further 
down  the  road  so  I  can  stay  current. 

3.  Did  any  of  your  instructors  encourage  you  to  contribute  content  to  the  course  materials? 

( )  No;  nobody  asked  for  student  input.  It  was  assumed  that  instructors  do  the  teaching, 
and  students  do  the  learning. 

( )  Sort  of;  more  often  than  not,  instructors  did  not  ask  for  input,  but  we  were  encouraged 
to  contribute  if  we  offered. 

( )  Yes;  instructors  generally  recognized  students'  specific  areas  of  expertise  and  actively 
found  ways  to  fold  our  input  into  readings  and/or  classroom  discussion. 

4.  Did  any  of  your  instructors  encourage  you  (or  demonstrate  how)  to  find  resources  in  addition 
to  him  or  herself  to  learn  more  about  a  topic  discussed  in  class? 

( )  No;  instructors  generally  saw  themselves  as  the  leading  authority  on  the  topic  they 
were  teaching. 

( )  Sort  of;  instructors  didn't  discourage  independent  research,  but  it  also  wasn't  important 
for  doing  well  in  class. 
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( )  Yes;  instructors  actively  encouraged  us  to  do  our  own  research  by  using  professional 
forums,  AKO,  CALL,  or  other  resources,  and  showed  people  where  to  find  things  if 
they  didn't  already  know. 

5.  When  you  searched  for  current  information  to  complete  class  assignments  or  to  conduct 
independent  research,  how  would  you  rate  the  importance  of  the  following  resources: 

Professional  forums  in  the  Battle  Command  Knowledge  System  (BCKS)  Leader  Network 

( )  Extremely  important  -  This  was  the  only  type  of  resource  I  used 
( )  Somewhat  important  - 1  used  this  type  of  resource  more  than  others 
( )  Equal  importance  - 1  used  this  type  of  resource  as  much  as  I  did  the  others 
( )  Not  too  important  - 1  used  other  types  of  resources  more  than  this  one 
(  )  Not  at  all  important  - 1  never  used  this  type  of  resource 
( )  Didn't  search  for  current  information 

Army  knowledge  repositories  (e.g.,  AKO,  CALL,  etc.) 

( )  Extremely  important  -  This  was  the  only  type  of  resource  I  used 
( )  Somewhat  important  - 1  used  this  type  of  resource  more  than  others 
( )  Equal  importance  - 1  used  this  type  of  resource  as  much  as  I  did  the  others 
(  )  Not  too  important  - 1  used  other  types  of  resources  more  than  this  one 
( )  Not  at  all  important  - 1  never  used  this  type  of  resource 
( )  Didn't  search  for  current  information 

Non- Army  resources  (e.g.,  Google,  Early  Bird,  etc.) 

( )  Extremely  important  -  This  was  the  only  type  of  resource  I  used 
( )  Somewhat  important  - 1  used  this  type  of  resource  more  than  others 
( )  Equal  importance  - 1  used  this  type  of  resource  as  much  as  1  did  the  others 
( )  Not  too  important  - 1  used  other  types  of  resources  more  than  this  one 
( )  Not  at  all  important  - 1  never  used  this  type  of  resource 
( )  Didn't  search  for  current  information 

Additional  Comments  regarding  the  Course  Content  (optional): 

Section  II  -  Course  Administration 

6.  What  was  the  primary  means  you  used  to  access  course  materials? 

( )  Blackboard 
( )  SharePoint 
( )  g:  drive 
( )  Email 

( )  Memory  stick/thumb  drive 
( )  Printed  documents  from  the  instructor 
()  Other  [  ] 
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7.  What  was  the  primary  means  you  used  to  coordinate  with  other  students? 

( )  Blackboard 
( )  SharePoint 
( )  Email 
( ) Phone 

( )  Face-to-face  meetings 
( )  Other  [  ] 

8.  How  often  did  you  access  Blackboard  prior  to  a  class  meeting  in  order  to  prepare  by  reading 
or  reviewing  lecture  materials  ahead  of  time? 

( )  Never;  our  class  content  wasn't  posted  in  Blackboard/there  was  always  a  better  way 
than  Blackboard  to  get  to  course  content. 

( )  Sometimes;  if  the  course  content  was  actually  on  Blackboard  and  I  had  the  time. 

( )  Often;  course  content  was  usually  on  Blackboard  and  I  tried  to  make  time  to  get  to  it 
before  class. 

( )  Always;  I  could  count  on  content  being  on  Blackboard  and  actively  used  access  to  it  to 
prepare. 

(  )  N/A;  I  didn't  prepare  for  class  by  reading  ahead  or  reviewing  lecture  materials. 

9.  Was  there  someone  among  the  students  in  your  staff  group  who  used  SharePoint  to  coordinate 
student  efforts  (e.g.,  group  assignments,  due  date  lists,  class  calendar,  etc.)? 

( )  No;  we  all  pretty  much  fended  for  ourselves  when  trying  to  succeed  in  class. 

( )  No;  we  had  a  student  leader,  but  he/she  didn't  use  SharePoint  to  coordinate  our  efforts. 
( )  Yes;  we  had  a  staff  group  student  leader  appointed  by  the  instructor  to  help  out. 

()  Yes;  a  staff  group  student  leader  emerged  within  the  first  few  weeks  of  class. 

How  difficult  was  it  to  access _ from  outside  the  schoolhouse? 

1 0.  Blackboard 

( )  Extremely  difficult;  I  stopped  trying  altogether. 

( )  Somewhat  difficult;  I  only  did  it  when  I  absolutely  had  to. 

( )  I  wouldn't  know;  I  didn't  try  to  access  it  from  outside  the  schoolhouse. 

( )  Not  too  difficult;  I  had  no  more  difficulty  with  accessing  this  than  one  would  expect 
( )  Extremely  easy;  I  never  had  any  problem  accessing  this  from  outside  the  schoolhouse 

1 1 .  SharePoint 

( )  Extremely  difficult;  I  stopped  trying  altogether. 

( )  Somewhat  difficult;  I  only  did  it  when  I  absolutely  had  to. 

( )  I  wouldn't  know;  I  didn't  try  to  access  it  from  outside  the  schoolhouse. 

( )  Not  too  difficult;  I  had  no  more  difficulty  with  accessing  this  than  one  would  expect 
( )  Extremely  easy;  I  never  had  any  problem  accessing  this  from  outside  the  schoolhouse 
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What  would  you  do  to  make  the  system  easier  to  access? 

12.  How  often  did  your  common  core  instructors  do  the  following: 

Provide  clear  guidance  regarding  their  expectations  and  performance  evaluation  criteria? 

( )  Every  instructor  did  this 
( )  More  instructors  did  this  than  didn't  do  this 
( )  About  as  many  instructors  did  this  as  didn't  do  this 
( )  Fewer  instructors  did  this  than  didn't  do  this 
()  No  instructor  did  this 

Provide  constructive  and  informative  feedback  on  your  performance? 

( )  Every  instructor  did  this 
( )  More  instructors  did  this  than  didn't  do  this 
( )  About  as  many  instructors  did  this  as  didn't  do  this 
( )  Fewer  instructors  did  this  than  didn't  do  this 
()  No  instructor  did  this 

Present  learning  challenges  that  were  difficult  but  not  overwhelming? 

( )  Every  instructor  did  this 
( )  More  instructors  did  this  than  didn't  do  this 
( )  About  as  many  instructors  did  this  as  didn't  do  this 
(  )  Fewer  instructors  did  this  than  didn't  do  this 
()  No  instructor  did  this 

Assign  group  tasks  that  had  well-defined  tasks  and  roles  for  each  group  member? 

(  )  Every  instructor  did  this 
( )  More  instructors  did  this  than  didn't  do  this 
( )  About  as  many  instructors  did  this  as  didn't  do  this 
( )  Fewer  instructors  did  this  than  didn't  do  this 
()  No  instructor  did  this 

Engage  students  in  asking  questions  and  reflecting  on  class  materials? 

( )  Every  instructor  did  this 
( )  More  instructors  did  this  than  didn't  do  this 
( )  About  as  many  instructors  did  this  as  didn't  do  this 
(  )  Fewer  instructors  did  this  than  didn't  do  this 
( )  No  instructor  did  this 
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13.  During  the  common  core  course,  how  successful  did  you  feel  in  leading  your  own  learning 

process? 

( )  Fully  capable;  I  had  all  of  the  tools  and  access  I  needed  to  accomplish  my  learning 
goals. 

( )  Somewhat  capable;  I  felt  I  was  successful  in  reaching  many  of  my  learning  goals,  but 
I  sometimes  lacked  the  time  to  accomplish  everything  I  wanted  to. 

( )  Not  sure;  I  was  never  really  given  the  chance  to  lead  my  own  learning. 

( )  Not  very  capable;  I  don't  feel  I  had  the  kind  of  access  to  learning  resources  that  I 
needed  to  succeed. 

( )  Not  at  all  capable;  I  spent  so  much  time  just  trying  to  figure  out  Blackboard  and 
SharePoint,  that  I  had  little  opportunity  to  take  charge. 

( )  N/A;  I  don't  feel  students  should  lead  their  own  learning  -  that's  why  they  have 
instructors. 

14.  How  accessible  did  you  find  the  technical  support  to  be? 

( )  What  tech  support?  I  didn't  know  there  was  tech  support. 

( )  I  knew  how  to  find  the  tech  support  team,  but  actually  getting  a  hold  of  someone  was 
unpredictable;  on  more  than  one  occasion  I  had  to  wait  longer  than  I  would  have  liked. 

(  )  I  wouldn't  know;  I  knew  there  was  a  tech  support  team,  but  I  never  tried  to  contact 
anybody. 

( )  I  generally  didn't  have  trouble  finding  or  connecting  with  someone  in  tech  support;  if  I 
ever  had  to  wait,  it  wasn't  a  big  deal. 

( )  Finding  and  connecting  with  tech  support  was  no  problem  at  all;  I  always  got 
somebody  when  I  needed  them. 

15.  How  useful  did  you  find  the  technical  support  to  be? 

( )  Not  at  all  useful;  I  had  to  find  other  ways  to  fix  my  problems." 

(  )  They  solved  technical  problems  ok,  but  if  the  tech  support  was  better,  the  kinds  of 
problems  I  had  wouldn't  exist. 

( )  I  wouldn't  know;  I  never  asked  for  tech  support. 

( )  I  had  a  pretty  good  experience;  the  system  worked  well  enough  and  when  problems 
did  come  up,  they  were  usually  fixed  the  first  time  around. 

( )  They're  the  best;  the  system  worked  great  and  whenever  I  needed  technical  assistance, 
it  was  provided  quickly,  effectively,  and  professionally. 

Additional  Comments  regarding  the  Course  Administration  (optional): 
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Section  III  -  Overall  Learning  Experience 

16.  How  motivated  did  you  feel  to  learn  the  subjects  in  the  common  core  course? 

( )  Highly  motivated;  the  course  content  spoke  to  my  interests  and/or  the  instructors  were 
engaging. 

( )  Generally  motivated;  more  often  than  not,  the  course  content  spoke  to  my  interests 
and/or  the  instructors  were  engaging. 

( )  Generally  not  motivated;  the  course  content  usually  did  not  speak  to  my  interests 
and/or  the  instructors  were  not  engaging. 

( )  Barely  motivated;  the  course  content  was  irrelevant  and/or  instructors  seemed  poorly 
qualified  or  burned  out. 

17.  What  do  you  think  is  the  primary  reason  that  ILE  courses  are  being  administered  via 
Blackboard  and  SharePoint? 

( )  The  Army  wants  to  save  money. 

(  )  The  Army  leadership  likes  Blackboard  and  SharePoint  and  decided  everyone  has  to 
use  it. 

( )  The  Army  thinks  using  Blackboard  and  SharePoint  is  in  the  best  interests  of  leader 
education. 

(  )  I  couldn't  say  how  the  decision  to  use  Blackboard  and  SharePoint  was  made. 

(  )  Other  [  ] 

18.  The  instruction  I've  received  on  Blackboard  supports  my  use  of  Blackboard  in  ILE,  so  far. 

( )  Strongly  Agree 
( )  Agree 
( )  Undecided 
( )  Disagree 
( )  Strongly  Disagree 

19.  When  I  use  Blackboard,  connecting  to  the  internet  has  not  been  a  recurring  problem. 

( )  Strongly  Agree 
( )  Agree 
( )  Undecided 
( )  Disagree 
( )  Strongly  Disagree 

20.  The  Blackboard  Learning  System  will  enhance  my  ILE  education. 

( )  Strongly  Agree 
( )  Agree 
( )  Undecided 
( )  Disagree 
( )  Strongly  Disagree 
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21 .  My  ILE  instructors  are  knowledgeable  on  the  Blackboard  Learning  System. 
{Choose  one} 

( )  Strongly  Agree 
( )  Agree 
( )  Undecided 
( )  Disagree 
( )  Strongly  Disagree 


22.  What  would  you  like  the  CGSC  leadership  to  know  about  any  aspect  of  the  Blackboard  or 
SharePoint  System(s)? 


The  following  questions  are  strictly  for  demographic  information  only. 


23.  What  CGSC  Class  are  you  in? 

( )  AY  007-001 
( )  AY  006-002 
( )  AY  006-002  (Pilot) 


24.  My  CGSS  Section  is 

()1 

()2 

( )  3  (PILOT) 

04 
05 
07 
08 
09 
0  10 
Oil 
012 
()13 
015 
016 
018 
019 
O  20 


Staff  Group 
( )  A 
OB 
()C 
()D 


134 


This  completes  the  survey. 

If  you  would  like  more  information  regarding  the  Lifelong  Learning  Center,  copy  and  paste  the 
link  below  into  the  address  bar  on  the  next  page. 

http://www.tradoc.army.mil/paoAVeb_specials/lifelong_leaming/intro.htm 
Again,  thank  you  for  participating  in  this  survey.  Your  time  and  responses  are  valued. 
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APPENDIX  C 
ACRONYMS 


AKO 

Army  Knowledge  Online 

BCKS 

Battle  Command  Knowledge  System 

CBT 

CGSC 

CGSOC 

CGSS 

computer-based  training 

Command  and  General  Staff  College 
Command  and  General  Staff  Officers’  Course 
Command  and  General  Staff  School 

DA 

DMOSQ 

DOET 

Department  of  the  Army 

Duty  military  occupational  specialty  qualified 
Directorate  of  Educational  Technology 

FSDO 

Faculty  and  Staff  Development  Office 

ILE 

Intermediate-Level  Education 

KM 

knowledge  management 

LLC 

LVN  LLI 

Lifelong  Learning  Center 

Leavenworth  Lifelong  Learning  Initiative 

METL 

MOSQ 

Mission  essential  task  list 

Military  occupational  specialty  qualified 

QAO 

Quality  Assurance  Office 

SADL 

SOP 

School  of  Advanced  Distance  Learning 
Standard  Operating  Procedure 

TASS 

TRADOC 

Total  Army  School  System 

U.S.  Army  Training  &  Doctrine  Command 

WBT 

Web-based  training 
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